Commit Graph
485 Commits
Author SHA1 Message Date
Aleksei Sviridkin e4d059cedd fix(etcd): require an even spread of members over nodes by default (#4986)
## What this PR does

The tenant etcd chart spreads members with `maxSkew: 1` on
`kubernetes.io/hostname` and `whenUnsatisfiable: ScheduleAnyway`, the
default since #119. `ScheduleAnyway` only adds to the score of a node
and other scores can outweigh it, so two members of one etcd can still
start on the same node, where a node-local volume then keeps them. This
PR makes the default `DoNotSchedule` with `nodeTaintsPolicy: Ignore`.
The branch for `_cluster.scheduling.globalAppTopologySpreadConstraints`
is untouched, so a platform that sets its own policy renders exactly as
before, and the soft pod anti-affinity from #3962 stays.

Seen on a three-node cluster running v1.6.4, whose `etcd-cluster.yaml`
and etcd-operator v0.5.5 are the same as in v1.6.5. A new tenant etcd
got the Pods of its first two members on one node, created 26 seconds
apart, and the third on another node 18 seconds later. The Pods carry
the `ScheduleAnyway` constraint and were scheduled by
`linstor-scheduler`, whose profile adds the LINSTOR extender's
prioritize score to the same sum with weight 5. The data volumes are on
a node-local StorageClass with `WaitForFirstConsumer` and
`allowRemoteVolumeAccess: "false"`, so the two members cannot leave that
node, and losing it takes the quorum. A second tenant etcd created there
the next day came up the same way. On a test cluster four tenant etcds
created within one week each had two members on one node, and we moved
one member of each by hand.

`DoNotSchedule` does not stop an etcd with more members than nodes from
starting when every node can run its Pods, unlike a required
anti-affinity, because the skew is counted between the nodes that exist.
kube-scheduler rejects a node only when the matching Pods on it plus the
new one exceed the node with the fewest by more than `maxSkew`
([filtering.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/filtering.go#L334-L352)).
One node takes all three members, two nodes take two and one, and three
or more nodes take one each. A member whose only allowed node cannot
take it, for example because the storage pool there is full, now waits
Pending instead of joining another member.

`nodeTaintsPolicy: Ignore` is the Kubernetes default and is written out
on purpose. A cordoned node carries `node.kubernetes.io/unschedulable`,
and `Honor` would take it out of the count
([common.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/common.go#L50-L54)),
so a member created during a drain would start next to another member
and stay there on a node-local volume. With `Ignore` it waits Pending
until the node is back, while the other two keep quorum. The cost is
that a node no member can run on, such as a tainted control plane,
counts as an empty node too, so on a cluster with such a node each
schedulable node takes at most one member and the members beyond that do
not start. #4947 makes the same choice for the SeaweedFS masters and
checked both policies in kwok. A platform that wants the old behaviour
can set `scheduling.globalAppTopologySpreadConstraints` to
`ScheduleAnyway`, which the other charts that read it pick up as well,
and a new test case pins that the chart renders such a policy as
written.

On upgrade nothing is recreated. The etcd-operator adopts a changed
`spec.topologySpreadConstraints` into `status.observed` once the cluster
has reached its current target, or its progress deadline has passed
([etcdcluster_controller.go#L203-L214](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L203-L214)).
It copies the value from there into the spec of an `EtcdMember` only
when it creates one, at bootstrap
([#L446](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L446))
and on scale-up
([#L851](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L851)),
and nothing in the operator writes it into an existing `EtcdMember`. The
member controller creates a Pod only when it is missing and builds it
from that member's own copy
([etcdmember_controller.go#L499-L512](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L499-L512),
[#L545-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L545-L549),
[#L860](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L860)).
So running members keep `ScheduleAnyway`, and so does a member whose Pod
is evicted, deleted or replaced after a terminal phase
([#L164-L186](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L164-L186)),
since the new Pod is built from the same copy. Only members created
after the upgrade get `DoNotSchedule`, which means a scale-up, a
crash-loop replacement or a member whose `EtcdMember` was deleted. No
member is rolled, neither one at a time nor all at once. This matches
the field docs
([etcdcluster_types.go#L542-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdcluster_types.go#L542-L549),
[etcdmember_types.go#L112-L116](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdmember_types.go#L112-L116)).
The code paths are the same in v0.5.8, and in v0.5.5, which v1.6.4 and
v1.6.5 ship, except that v0.5.5 does not yet replace a Pod left in a
terminal phase.

Members that already share a node therefore stay there. One of them
moves only when a new `EtcdMember` takes its place, and the constraint
then sends the replacement to the node without a member. That is how we
moved the members on the test cluster, with `DoNotSchedule` set on the
`EtcdCluster` by hand. The operator's docs say a scheduling change can
be applied by deleting one Pod at a time
([concepts.md#L272](https://github.com/cozystack/etcd-operator/blob/v0.5.6/docs/concepts.md#L272)),
but the recreated Pod gets the member's old copy, so that does not move
it.

The release note now declares two cases from review. In the first, an
etcd created while the cluster had fewer nodes than members has two or
three members on one node, and once a node is added a recreated Pod of
one of them stays Pending, since `maxSkew` rejects its own node and its
node-local volume rules out the others. Deleting that member's
`EtcdMember` while the other members keep quorum replaces it on a node
without a member. In v0.5.5 the member finalizer removes the member from
etcd through the remaining members before the object goes
([`handleDeletion`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L195-L231),
[`removeMemberFromEtcd`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L243-L310)),
its Pod and its `data-<member>` PVC are owned by it and deleted with it
([#L403](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L403),
[#L490](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L490)),
and the cluster controller waits for the deletion, then adds a member
under a new name with the constraints from `status.observed`
([etcdcluster_controller.go#L254-L312](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L254-L312),
[`scaleUp`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L820-L851)).
Its new PVC binds where its Pod lands, and under `DoNotSchedule` only a
node without a member passes. The operator documents the same steps for
a broken member, along with the limit that `MemberRemove` cannot run
without quorum
([operations.md#L208-L231](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L208-L231)),
and v0.5.6, which main and `release-1.6` ship now, has the same path.
The upstream chart of the operator installs a policy that denies
`EtcdMember` deletion without the
`etcd-operator.cozystack.io/allow-deletion=true` annotation
([member-deletion-policy.yaml](https://github.com/cozystack/etcd-operator/blob/v0.5.5/charts/etcd-operator/templates/member-deletion-policy.yaml)),
and the `etcd-operator` package here does not install it.

On a three-node cluster running v0.5.5 a tenant etcd had two members on
one node with `DoNotSchedule` in effect. We deleted the `EtcdMember` of
one of the two while the other members kept quorum. The replacement's
Pod and its new node-local PVC were scheduled on the node without a
member, the member joined, and the `EtcdCluster` stayed `Available` with
quorum the whole time, polled every 5 seconds.

In the second case `nodeTaintsPolicy: Ignore` counts a node the Pods
cannot tolerate as one with zero members, so on a cluster with such a
node and fewer schedulable nodes than members the members beyond the
schedulable nodes do not start. The way out is the soft global policy in
the platform `Package`, set before the upgrade.

```yaml
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
  name: cozystack.cozystack-platform
spec:
  components:
    platform:
      values:
        scheduling:
          globalAppTopologySpreadConstraints: |
            topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: ScheduleAnyway
```

The platform chart writes `scheduling` into `_cluster` of the
`cozystack-values` Secret
([apps.yaml#L292-L295](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/core/platform/templates/apps.yaml#L292-L295)),
the tenant chart copies `_cluster` into the Secret of each tenant
namespace
([namespace.yaml#L109-L120](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/apps/tenant/templates/namespace.yaml#L109-L120)),
and this chart renders the value as written, which the new test case
pins. Rendering the platform and tenant charts with the value above, on
main and on `release-1.6`, gives the same string in both Secrets. The
operator adopts a changed spec into `status.observed` only once all
members are Ready or the progress deadline has passed
([etcdcluster_controller.go#L196-L213](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L196-L213)),
and it clears the deadline when the cluster converges
([#L1448-L1456](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L1448-L1456)).
A new etcd stuck short of its members adopts the policy once its first
deadline passes, and deleting the `EtcdMember` of the Pending member
then brings a replacement that starts. An etcd that had converged before
one of its replacements went Pending adopts the policy only after the
deadline is forced as the operator docs describe
([operations.md#L197-L206](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L197-L206)),
which is why the release note says to set it before the upgrade.

#4817 moves this block into `_etcd.tpl` in cozy-lib without changing it,
so whichever of the two lands second has to carry the two lines over.

`tests/topology-spread_test.yaml` now pins the whole default constraint,
and a new case pins that a `ScheduleAnyway` policy from
`globalAppTopologySpreadConstraints` renders as written, without
`nodeTaintsPolicy`. The existing zone override case is unchanged.
Against the template on main the suite fails only the default case, 1
failed and 25 passed. Switching the default to `Honor`, to
`ScheduleAnyway` or to `maxSkew: 2` fails the default case, and adding
`nodeTaintsPolicy` to the override branch fails the new case.

```console
$ docker run --rm -v "$PWD":/apps -w /apps/packages/extra/etcd helmunittest/helm-unittest:4.2.3-1.1.2 .
Charts:      1 passed, 1 total
Test Suites: 9 passed, 9 total
Tests:       26 passed, 26 total
```

On main the same command gives 25 tests in 9 suites, with helm-unittest
1.1.2 as CI pins it. With default values `helm template` differs from
main in the two lines and in the comment above the affinity block, and
with a global policy set only in that comment. The docs commit changes
the `affinity` description, from which `make generate` rewrites the
README, `values.schema.json` and the `ApplicationDefinition` with
nothing but that text, and the suite stays at 26 tests. I have not
rolled this chart change out on a cluster.

The spread block and the test file are the same on release-1.6 and
release-1.7. On release-1.6 the template has no affinity block yet, so
the fix commit cherry-picks without conflicts and the docs commit has
nothing to change there. That backport is a separate PR.

### Screenshots

Not a UI change.

### Downstream repositories

One changed and one added line in
`packages/extra/etcd/templates/etcd-cluster.yaml` and its test suite,
and new text for the `affinity` description and the comment above the
affinity block. The description reaches `values.schema.json`, the README
and the `ApplicationDefinition` as text only, with no key, type or
default change, and the website rebuilds the etcd reference page from
the README on release. The website mentions
`globalAppTopologySpreadConstraints` only as a platform setting, which
stays as it is, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(etcd): tenant etcd members now spread over nodes with whenUnsatisfiable DoNotSchedule by default, so two members of one etcd no longer start on the same node while another node has none. Members that exist are not moved, only members created after the upgrade get the new constraint.

An etcd created while the cluster had fewer nodes than members puts two or three members on one node. Once a node is added, such a member whose Pod is recreated (drain, eviction, graceful node shutdown) stays Pending, because maxSkew rejects its own node and its node-local volume rules out the others. With two nodes grown to three the etcd runs two of three members and its PodDisruptionBudget blocks draining the other nodes. With one node grown to three a reboot of that node brings back one member of three and quorum is lost. A stranded member is replaced by deleting its EtcdMember while the other members keep quorum. Its Pod and PVC go with it, and the replacement gets a new Pod and PVC on the node without a member. Right after adding nodes, doing this for one co-located member at a time while all members are Ready avoids the case.

With nodeTaintsPolicy Ignore a node the etcd Pods cannot tolerate, such as a tainted control plane, counts as a node with zero members, so no node can take a second member. On a cluster with such a node and fewer schedulable nodes than members, a new etcd starts only as many members as there are schedulable nodes, one of three on a single schedulable node. An existing etcd there keeps its members, but a member it replaces stays Pending the same way.

Either case is avoided by setting scheduling.globalAppTopologySpreadConstraints in spec.components.platform.values of the cozystack.cozystack-platform Package to a topologySpreadConstraints entry with maxSkew 1, topologyKey kubernetes.io/hostname and whenUnsatisfiable ScheduleAnyway before the upgrade. Every app chart that reads this value applies it, two members of a new etcd can share a node again, and the operator gives it only to members created after it adopts the changed EtcdCluster spec into status.observed. Once status.observed.topologySpreadConstraints of an EtcdCluster shows ScheduleAnyway, a member of it that stays Pending is replaced by deleting its EtcdMember.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **Bug Fixes**
- Default etcd placement now requires even spreading across hosts and
ignores node taints when evaluating placement.
- Configurable cluster-wide spreading continues to use a flexible
scheduling policy when selected.
- **Documentation**
- Clarified how affinity and topology spreading work together, when
required anti-affinity or node-local storage can leave members pending,
and that affinity changes take effect as members are replaced.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:39:53 +03:00
Aleksei Sviridkin 3fce040268 Merge branch 'main' into fix/etcd-strict-member-spread
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:49 +03:00
Aleksei Sviridkin d1f95e3212 Merge branch 'main' into fix/etcd-defrag-small-requests
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:32 +03:00
Aleksei Sviridkin b6f5a4fa35 feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does

Adds a default scrape class `application-identity` to both VMAgents: the
cluster-wide one in `monitoring-agents` and the per-tenant one in
`monitoring`. Every series scraped from a pod that carries the lineage
labels `apps.cozystack.io/application.{group,kind,name}` gets them as
`apps_cozystack_io_application_{group,kind,name}`.

Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Design 4a)
needs to tell which application an exporter series belongs to. Today
that is only possible indirectly, through pod names.

How the labels are protected against forgery by an exporter:

- At the target stage, the class always sets all three labels: to the
pod's real lineage value, or to a placeholder `-` when the pod has none
(`-` is not a valid Kubernetes label value). With `honorLabels: false`,
any copy the exporter publishes therefore collides with the target label
and is renamed to `exported_*`.
- At the metric stage, the class drops every `exported_*` copy.
- The placeholder is removed by the VMAgent's global remote-write
relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent
adds `up` and `scrape_*` with the target labels after metric relabeling,
so only the remote-write stage sees those series. Targets without
lineage therefore keep exactly the label set they have today on every
series, including `up` and `scrape_*`.
- With `honorLabels: true` the exporter's value wins and no relabel rule
can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and
its series describe Ingress objects in any namespace, so the
controller's identity would misattribute traffic; its scrape now drops
the identity labels explicitly, from the target (so `up` and `scrape_*`
stay unchanged) and from the scraped samples (so an exporter's copy
cannot survive). Switching it to `honorLabels: false` was not done,
because that renames `namespace` and breaks the existing ingress
dashboards.

#### How it was tested

- helm-unittest for `monitoring-agents`, `monitoring` and `ingress`.
These tests check the rendered config only; the relabeling behaviour was
verified offline by running vmagent v1.136.0, built from source, against
the rendered rules: `up` and `scrape_*` carry no identity for targets
without lineage, and the placeholder appears on no series.
- The rendered `scrapeClasses` validate against the vendored VMAgent CRD
schema.
- The rendered rules were exercised on a locally built vmagent v1.136.0
(the version the bundled operator deploys) writing to a local
VictoriaMetrics: with and without lineage, with and without a forging
exporter, `honorLabels` true and false, and the ingress-nginx endpoint
rules. Checked on exporter samples and on `up`, `scrape_samples_scraped`
and `scrape_series_added`; the previous rules reproduce the `-`
placeholder on the generated series of targets without lineage, the new
ones leave no placeholder on any series.
- Verified on a test cluster with the earlier revision that removed the
placeholder at the metric stage (the move to remote-write relabeling is
not yet verified on a cluster): series carry the application labels;
metrics read for a Redis instance through the per-application proxy root
contained only that instance's pods; vmagent logs show no errors.

#### Known limitations

- Scrapes with `honorLabels: true` other than ingress-nginx are not
protected against an exporter that publishes the identity labels itself.
Today these are the vendored qdrant and harbor ServiceMonitors; their
exporters are platform images with fixed label names, and their series
describe the application's own pods.
- A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes
entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth
settings from a scrape class (operator v0.68.4); no VMProbe exists in
the platform today.
- A scrape object's own relabel rules run after the class and can
overwrite the labels; tenants cannot create VictoriaMetrics operator
objects, so this is platform-controlled.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes templates and tests in `packages/system/monitoring-agents`,
`packages/system/monitoring` and `packages/extra/ingress` without
adding, renaming or removing a package and without touching any
`values.yaml` or schema. The telemetry metrics the telemetry server
reads (`cozy_cluster_info`, `cozy_application_count`) are produced by
the operator, not by these scrapes. The proposal text in
`cozystack/community` (#94, Design 4a) should be corrected to name
qdrant and harbor as further `honorLabels: true` scrapes; that is not a
trigger-map item.

### Release note

```release-note
feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Monitoring metrics now use application identity labels derived from
Kubernetes pod metadata for covered scrape targets. Exporter-supplied
duplicates and placeholder values are removed, including from `up` and
`scrape_*` metrics.
* Ingress metrics no longer retain application identity labels at either
target or metric relabeling stages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:03:57 +03:00
Aleksei Sviridkin 238b3ee491 fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but
only the `s3_lifecycle` job of its plugin worker applies it, and that
job is scheduled by the admin server. Cozystack deployed neither, so a
lifecycle rule returned 200 and no object ever expired, on the platform
backup bucket and on tenant buckets alike. That also made the documented
way to bound backup storage a no-op.

This PR adds the admin server and a worker to Simple and MultiZone
SeaweedFS instances, behind a new `lifecycle.enabled` value that is
**off by default**. A Client instance is unchanged, because its rules
are enforced where the instance it connects to runs. The worker
registers for `s3_lifecycle` only, so vacuum, balancing and erasure
coding are not started as a side effect. The admin keeps its job state
on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default
when that is empty; `volume.storageClass` is not reused, because in
MultiZone it can name a class local to one zone or node. The admin UI is
protected by a generated password, because a tenant namespace admits
traffic from the whole cluster. The dashboard Role lists the two new
WorkloadMonitors.

Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart
renders no `[grpc.s3]` section, so the s3 gateway served its internal
gRPC in plaintext while the worker dials it with mTLS, and every
lifecycle delete failed the TLS handshake. The section is backported
from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart
patch, applied with `--forward` so the next `make update` stops at the
point where the patch must be dropped instead of silently reversing it.
The s3 pods are restarted once so they load the section. This also
repairs IAM cache propagation between s3 replicas, which failed in the
same way, and it applies whether or not lifecycle is enabled.

The data-plane release now retries a failed install or upgrade in place
(`RetryOnFailure`) instead of rolling it back. With the previous default
remediation and unlimited retries, an upgrade that waited out its
timeout, for example on a tenant quota with no room for the admin
volume, rolled back and retried forever, restarting the s3 pods twice
per cycle. The worker's hostname anti-affinity is relaxed to a
preference so a rollout finishes on a single-node cluster.

#### Why lifecycle is off by default

The daily walk that SeaweedFS uses to expire objects sends its deletes
without an object identity, and the s3 server skips its compare-and-swap
when no identity is given; it does not re-check the object's age either.
A key overwritten between the walk's listing and the delete loses its
new content. This path is unchanged from 4.45 through 4.52 and on
upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`,
`identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup
buckets write unique keys and are barely exposed; a tenant bucket that
rewrites a fixed key is. On top of that, turning enforcement on would
start applying, on upgrade, every rule that was stored and ignored until
now. Both are retention changes an operator should choose, so nothing
changes for existing instances until `lifecycle.enabled` is set. Making
it the default again is left to a separate PR that carries object
identity on the walk path; upstream report:
https://github.com/seaweedfs/seaweedfs/issues/12451.

#### Turning lifecycle on and off

When `lifecycle.enabled` is set, rules that have been stored but ignored
start applying within minutes, and objects they have already expired are
deleted on that first pass, so review existing bucket lifecycle rules
first. Set `lifecycle.storageClass` before enabling, and prefer a
replicated class: the class is part of the admin StatefulSet's volume
claim template, which cannot change, so changing it while lifecycle is
on makes every upgrade of the instance fail until it is reverted. Moving
the volume means disabling lifecycle, deleting the
`admin-data-seaweedfs-admin-0` PVC, changing the class and enabling
lifecycle again. Each instance gains two small pods and a 1Gi PVC; when
the tenant quota has no room for them, the data-plane release keeps
retrying in place until it has.

When `lifecycle.enabled` is turned off again, the parent release removes
the admin credentials Secret while the data-plane release removes the
admin StatefulSet only on its own reconcile, shortly after. An admin pod
that restarts in that window fails to start on the missing Secret until
the StatefulSet is gone; nothing is lost, and the order is left as is.
The admin PVC stays until the instance is deleted, and a later re-enable
starts the admin with a new password.

Verified on a test cluster with the 4.45 chart: without the backport,
the worker found the expired objects and every delete failed with `tls:
first record does not look like a TLS handshake`; with this branch and
lifecycle enabled, a rule with a past expiration date removed the
matching objects and left the others, both on a manual run and on the
first scheduled pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up needed, not opened yet:
`operations/services/backup-classes.md` (v1.7 and next) advises an S3
lifecycle policy, which the bundled SeaweedFS applies only once
`lifecycle.enabled` is set; the SeaweedFS reference page gains
`lifecycle.enabled` and `lifecycle.storageClass` from the regenerated
README on the next release
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The terraform provider does not model the SeaweedFS application, so the
new values do not reach it.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added optional, daily S3 bucket lifecycle-rule processing for
supported SeaweedFS configurations. It is disabled by default; Client
instances rely on lifecycle rules configured on the connected SeaweedFS.
* Added configurable storage for lifecycle job state and dashboard
visibility for lifecycle components.
* Enabled the S3 gRPC server to use its internal TLS identity when
security is configured.

* **Bug Fixes**
* Improved storage-class handling so numeric-looking names remain
strings and unset classes are omitted where appropriate.
* Clarified backup cleanup guidance, including when lifecycle rules
apply and how to reclaim storage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:21:52 +03:00
Aleksei Sviridkin aa97451bc2 fix(platform): leave tenant-root's ingress empty when the root tenant has none (#5226)
## What this PR does

Enabling SeaweedFS on the root tenant without ingress no longer hangs.
`cozystack-basics` wrote `_namespace.ingress: tenant-root` for the root
tenant whatever its `spec.ingress` said, so `seaweedfs-system` depended
on an `ingress-nginx-system` release that nothing creates, and the
platform backup bucket waited with it. Every other tenant already
derives the key in `packages/apps/tenant/templates/namespace.yaml` and
leaves it empty without ingress.

`cozystack-basics` cannot read the root Tenant, so the platform now
carries its ingress state the same way it carries monitoring: the new
platform value `ingress.rootEnabled` reaches the `cozystack-values`
Secret as `_cluster.ingress-enabled`, and `_namespace.ingress` is
written only when it is true. The default is `true`, which keeps the
value every existing cluster renders today, and a Secret written before
the key existed keeps it as well. An operator running the root tenant
without ingress sets `ingress.rootEnabled: false`, kept in sync with
`Tenant/root spec.ingress` by hand, as with `monitoring.rootEnabled`.

With the key empty, `seaweedfs-system` drops the dependency through the
check it already had (`eq $ingress .Release.Namespace`), and nothing
reads live cluster state for it.

An empty `_namespace.ingress` now reaches tenant-root and every tenant
inheriting from it, so the consumers that would break on it are adjusted
in the same PR. The API rejects `ingressClassName: ""` (checked with a
server-side dry run), and two consumers quote the value:

- `kubernetes`: the proxied ingress-nginx addon Ingress is not rendered
without an ingress; its Service still is.
- `monitoring`: Grafana's `spec.ingress` is left out, since
grafana-operator copies it into the Ingress verbatim.
- `seaweedfs`: the S3 Ingress is disabled instead of rendering with no
controller to serve it.

The `KamajiControlPlane` `network.ingress` block is unchanged on
purpose: its `hostname` is what the Kamaji provider uses as the control
plane endpoint, and Kamaji itself skips an empty class name. Harbor,
bucket, bootbox and alerta write the class unquoted, so an empty value
renders as an unset class, which the API accepts.

Tests:
`packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml`
pins the flag reaching `_cluster` both ways and with `ingress` set to
null;
`packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml`
pins `_namespace.ingress` for a missing, string, bool and `"false"`
flag, and that the other keys stay when ingress is off;
`packages/extra/seaweedfs/tests/ingress_dependency_test.yaml` pins the
dependency and the S3 Ingress for an own, empty and inherited ingress;
the kubernetes and monitoring suites pin the gated Ingress. Reverting
any of the gates, or flipping the missing-key default, fails the suites.

Not covered here:

- `_namespace.etcd` has the same hardcoded shape (#4783) and is left for
its own change.
- Toggling `Tenant/root spec.ingress` without changing
`ingress.rootEnabled` leaves the value stale, as
`monitoring.rootEnabled` does today.

### Screenshots

Not a UI change.

### Downstream repositories

The diff adds a key to `packages/core/platform/values.yaml`, which the
website documents by hand.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#800 documents `ingress.rootEnabled` in the
platform Package reference
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a root tenant without ingress no longer leaves seaweedfs-system waiting for an ingress-nginx-system release that is never created. Set the new platform value ingress.rootEnabled to false to match Tenant/root spec.ingress: false.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a root-tenant ingress setting, enabled by default, that can be
turned off.
* **Bug Fixes**
* Kubernetes and Grafana ingress resources are now rendered only when
ingress is configured.
  * SeaweedFS S3 ingress is enabled only when ingress is configured.
* Ingress-dependent components now respect the root-tenant setting,
while other platform services remain available when root ingress is off.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 15:06:12 +03:00
Myasnikov Daniil 6a93765351 fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix
seaweedfs-cosi-driver v0.3.1 grants access by rewriting the legacy
/etc/iam/identity.json. Each S3 gateway migrates that file to
per-identity files and renames it away whenever it changes, so a grant
written while a gateway is migrating is lost: the BucketAccess reports
accessGranted and its Secret is written, but every gateway rejects the
key with InvalidAccessKeyId and never recovers. Users added to a Bucket
in one change lose a key regularly. v0.4.0 also writes the per-identity
file on grant.

The rest of v0.4.0 does not reach the platform. The
readwrite-governance-bypass access policy is opt-in and no
BucketAccessClass uses it. The refusal to delete a bucket that still
holds locked objects applies only to object-lock buckets, and their
classes are Retain, so the sidecar never calls DriverDeleteBucket for
them.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 07:53:34 +00:00
Alexey Artamonov 67fcadf689 fix(seaweedfs): disable the S3 Ingress in a tenant without ingress
The S3 Ingress kept rendering with no ingress-nginx to serve it. Now
that seaweedfs-system no longer waits on ingress-nginx-system there, the
release completes and the unserved Ingress would stay behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov b26001b6f9 test(seaweedfs): pin the ingress-nginx dependency to the namespace's ingress
seaweedfs-system waits on ingress-nginx-system only when _namespace.ingress
names its own namespace. With tenant-root's key now empty when the root
Tenant runs no ingress, an empty value has to drop the dependency, and a
value inherited from a parent must not add it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:36 +03:00
Alexey Artamonov f5114ae04c fix(seaweedfs): keep the metadata database storage class a string
db.storageClass went unquoted into the values of the database release
and again into its CNPG Cluster. A class name made only of digits,
which Kubernetes allows, became a number that the Cluster rejects, and
a line break in the value could add keys to the database release
values. It is quoted at both steps.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:30:11 +03:00
Alexey Artamonov e6eb3345c6 fix(seaweedfs): escape pool and zone storage classes in the release values
Pool and zone storage classes were wrapped in quotes by hand, which
does not escape a quote or a backslash in the value: a quote closes the
string early and the data-plane release values stop parsing, with a
YAML error that does not name the field. They are now quoted like the
volume and admin classes.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 10:29:48 +03:00
Alexey Artamonov 7c9aa2fe10 fix(seaweedfs): hand storage class names to the data plane as strings
The volume and admin storage classes were written unquoted into the
values of the data-plane release. A class name made only of digits,
which Kubernetes allows, arrived there as a number, and a value with a
line break could add keys next to the field, past the schema that is
the tenant's input gate. Both are now quoted. lifecycle.storageClass,
new in this release, also carries the StorageClass name rules in its
schema, so a value that cannot name a class is refused when the
application is written instead of leaving the admin claim pending.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-10 23:43:06 +03:00
Alexey Artamonov 5cbb3a633e fix(seaweedfs): give the lifecycle admin volume its own storage class
volume.storageClass is picked for volume server data, and in a
MultiZone instance it can name a class whose volumes exist in one zone
or on one node. Reusing it for the admin volume would tie the admin to
that zone or node, so losing it would stop lifecycle enforcement for
the whole instance, a placement nobody chose for the admin. The admin
volume now takes lifecycle.storageClass, and the cluster default when
that is empty.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 16:04:24 +03:00
Alexey Artamonov 62b4213784 fix(seaweedfs): keep bucket lifecycle enforcement off unless enabled
The daily walk that expires objects sends its deletes without an object
identity, and the s3 server skips its compare-and-swap when none is
given (walker_dispatcher.go and identityMatches, unchanged from 4.45
through 4.52). It does not re-check the object's age either, so a key
overwritten between the walk's listing and the delete loses its new
content. Backup buckets write unique keys and barely meet that window;
a tenant bucket that rewrites a fixed key does.

Turning enforcement on by default would also start applying, on
upgrade, every rule that was stored and ignored until now. Both are
retention changes an operator should choose, so the admin server and
the worker now run only where lifecycle.enabled is set. On by default
can return once the walk carries identity.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 15:56:51 +03:00
Alexey Artamonov a6f4f557f3 fix(seaweedfs): let the dashboard read the lifecycle monitors
The dashboard reads an application's WorkloadMonitors by name through
its resource-map Role, so the admin and worker monitors the lifecycle
switch adds were invisible to the tenant: the components ran, but the
application page never listed them.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:10:29 +03:00
Alexey Artamonov a70a1cf83b fix(seaweedfs): retry the data-plane release instead of rolling it back
The admin volume and the two lifecycle pods are new demands on tenant
quota, so an upgrade into a namespace with no headroom can wait out its
timeout. Flux's default remediation then rolls the release back, and
with unlimited retries it does so on every attempt: each rollback drops
the [grpc.s3] section and the s3 restart annotation, each retry brings
them back, and the s3 pods restart twice per cycle until quota frees up.

RetryOnFailure keeps what was applied and retries it, as the parent
release that the apiserver builds for the application already does.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:03:43 +03:00
ZverGuy 699b5ce78e test(ingress): pin the position of the target-stage identity drop
A containment check passes even if a rule added after the drop sets the
identity labels again. Asserting both endpoints' full target relabel
lists fixes the drop as the last rule, so such a change fails the test.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-05 00:35:37 +07:00
ZverGuy 1018eaa3b6 docs(ingress): name the controller series that lose application identity
The comment said every ingress-nginx series describes served Ingress
objects. The metrics port also serves the controller's own series
(nginx process, reload status, go_*, up, scrape_*), and dropping the
identity on the whole endpoint removes it from those as well. Say so,
so the lost attribution reads as a decision rather than an oversight.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-05 00:27:02 +07:00
ZverGuy dab6035a97 fix(ingress): keep application identity off ingress-nginx scrape-health series
Dropping the identity in metricRelabelConfigs reaches only the scraped
samples. vmagent adds up and scrape_* with the target labels after
metric relabeling (VictoriaMetrics v1.136.0 lib/promscrape addAutoMetrics,
addRows with needRelabel=false), so those series still carried the
controller pod's identity. Dropping it from the target as well leaves
every ingress-nginx series with the label set it had before the scrape
class. The metric-stage drop stays: with honorLabels true an
exporter-supplied identity is not renamed and would otherwise survive.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-05 00:17:46 +07:00
Alexey Artamonov f36d864e99 fix(seaweedfs): run the worker that enforces bucket lifecycle rules
SeaweedFS accepts and stores a bucket lifecycle configuration, but only
the s3_lifecycle job of its plugin worker acts on it, and that job is
scheduled by the admin server. Neither ran here, so a rule returned 200
and no object ever expired, on the platform backup bucket and on tenant
buckets alike, and the documented way to bound backup storage did
nothing.

Both now run in Simple and MultiZone instances, behind lifecycle.enabled
(on by default). A Client instance has no data plane of its own; its
rules are enforced where the master it connects to runs. The worker
registers for s3_lifecycle alone, so enabling lifecycle does not also
start vacuum, balancing or erasure coding. The admin keeps its job state
on a small volume (on volume.storageClass when set), and its UI gets a
generated password, because a tenant namespace admits traffic from the
whole cluster and the server is unauthenticated without one. The
data-plane chart keeps both off unless its parent turns them on, so it
never starts an admin without that password.

Existing rules start applying minutes after the upgrade: the admin's
scheduler runs the first pass within one idle cycle of its lifecycle
lane, then once a day, and objects the rules have already expired are
deleted on that first pass.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:30:47 +03:00
ZverGuy 5a6086c7a1 test(monitoring): pin the endpoint count the ingress identity test relies on
The ingress assertion reads only the first endpoint of each scrape, so a
second endpoint added without the label drop would go unnoticed. Pinning
the count makes such an addition fail the suite.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:30:32 +07:00
ZverGuy 0e5557d0ba fix(ingress): keep application identity off ingress-nginx series
The vmagent scrape class stamps the scraped pod's application identity
on its series. ingress-nginx series describe the Ingress objects the
controller serves, in whatever namespace they live, so the controller's
own identity would attribute another tenant's traffic to the Ingress
application. And because these scrapes honor exporter labels, an
exporter-supplied identity would win over the class. Dropping the labels
leaves these series to the pod and namespace scopes.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:26:04 +07:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Yan Bondarenko 61289c7210 fix(etcd): size the defrag job for the client it runs
The hourly defrag pod runs the etcd-defrag client, which only sends
defragment calls to the members, yet it requested 200m CPU and 256Mi
from the tenant quota. On a sandbox tenant whose quota was mostly taken
by its own etcd and SeaweedFS, the job could not create a pod and the
defrag never ran. The client peaked at 5.1 MiB working set over a day
on a v1.6.5 cluster.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-25 17:09:16 +03:00
Yan Bondarenko 9f6fdd77c6 docs(etcd): update the affinity notes for the hard member spread
The affinity description and the comment above the affinity block
explained the soft default by a member that would stay Pending on its
node-local volume. Now that the spread is DoNotSchedule by default, the
spread keeps such a member Pending as well, once its node holds another
member and some other node holds none. Both texts now explain the soft
default by what is left, that a required rule forbids two members on
one node even when every node has one, and state the node-local case
for both rules.

The description also says that the operator copies affinity into each
EtcdMember when it creates it, so a recreated Pod keeps the rule its
member was created with. README, values.schema.json and the
ApplicationDefinition are regenerated from it.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-24 15:52:37 +03:00
myasnikovdaniil 432d02e20c fix(etcd): stop counting canceled watch streams as failed gRPC requests (#4985)
## What this PR does

The tenant etcd chart has its own copy of
`etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every
`grpc_code` other than `OK` as a failure. `grpc_server_handled_total`
counts a Watch stream once, when it ends, and the streams of a
kube-apiserver that goes away end with `Canceled`. So whenever an API
server on a tenant etcd stops, the Watch method of every member reads as
100 percent failed and the critical rule fires on all of them.

Both rules now count only
`Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`,
the list the etcd mixin has used since etcd-io/etcd#13127 and the one
the platform copy in
`packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The
grouping `BY (job, instance, grpc_service, grpc_method)`, the
thresholds, `for` and the annotations stay as they are, so the alerts
keep their labels.

Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was
created and deleted a few minutes later. vmalert had the critical alert
firing for `grpc_method="Watch"` on all three members with the value
100. At that point
`rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for
Watch with `Canceled`, and the only other non-OK codes were
`FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is
one call each. The same query an hour earlier returned nothing. The
platform copy runs in the same vmalert over the same series, and none of
the firing alerts came from it. Each alert also showed up seven times
there, once per tenant copy of these rules, because the root tenant's
vmalert loads the rules of every tenant namespace. This PR does not
change that.

`hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with
`helm template` and replays it through `vmalert-tool` v1.136.0 pinned by
digest, the same image #4963 uses.
`hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with
seven checks. Watch streams that end only with `Canceled` must not fire
at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25
percent must fire the critical rule at 7 minutes and both rules at 12,
with the labels and the summary pinned. Both together must fire only for
`Range`. Against the rules on main the suite fails 4 of the 7 checks,
all of them on Watch alerts with the value 100. Dropping `Unavailable`
from the list, or grouping by `job` and `instance` only, fails 3 checks
each.

```console
$ bats hack/etcd-grpc-alerts.bats
1..1
ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases
```

The same run on the rules from main prints `not ok 1` and the four
failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes
together with `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`,
`alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like
the migration suites it needs docker, and only the image pull touches
the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run
the new rules in vmalert on a live cluster.

The rules file is the same on release-1.6 and release-1.7. The commit
cherry-picks onto release-1.6 without conflicts, and that backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.

### Screenshots

Not a UI change.

### Downstream repositories

A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and
its test data. No values, schema, README, image or
`ApplicationDefinition` change, and the website does not mention these
rules, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Refined etcd gRPC failure alerts to target specific failure status
codes, reducing alerts for non-failure responses while preserving
existing thresholds and timing.
- Added coverage for canceled watch streams, unavailable range requests,
and combined failure scenarios, including warning and critical alert
evaluations.

- **Tests**
- Added integration validation for rendering etcd alert rules and
running them through the alert-testing tool, including failure handling
when prerequisites are unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-23 22:40:33 +05:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Yan Bondarenko f1f30b3071 fix(etcd): require an even spread of members over nodes by default
The default spread of the etcd members was ScheduleAnyway, which only
adds to the score of a node, and other scores can outweigh it. A new
etcd on a three-node cluster got its first two members on one node.
On a node-local StorageClass a member cannot leave its node, so losing
that node takes two of the three members and the quorum.

DoNotSchedule with maxSkew 1 still lets an etcd with more members than
nodes start, because the skew is counted between the nodes that exist.
One node takes all three members and two nodes take two and one.
nodeTaintsPolicy is Ignore, so a cordoned node still counts and a
member waits for it instead of landing next to another member.

The operator copies the constraints into an EtcdMember when it creates
it and builds the Pod from that copy, so members that exist keep the
old setting and only members created later get the new one.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-20 12:05:51 +03:00
Yan Bondarenko 1ca7c0d891 fix(etcd): stop counting canceled watch streams as failed gRPC requests
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted
every code other than OK. A Watch stream is counted once, when it ends,
and the streams of a kube-apiserver that stops end with Canceled. So
whenever an API server on that etcd goes away, the Watch method of
every member reads as 100 percent failed and the critical rule fires
on all of them.

The rules now count the codes the etcd mixin counts since
etcd-io/etcd#13127, the same list the platform copy in
monitoring-agents already uses. The grouping by method stays, so the
alert labels do not change.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-20 12:05:33 +03:00
Myasnikov Daniil f3486f4e3a docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed
them and left in the source tree, where no operator looks. They are on
the website, so a second copy here only drifts: edits kept landing in
docs/ after the website copy was made.

docs/vm-external-vlan.md is split rather than moved. Its section on
staging the reference CNI plugins is a contract that
hack/cni-plugins-staging-contract.bats asserts against the multus
Dockerfile, the daemonset manifest and the platform values, so that half
is package documentation and goes to packages/system/multus/README.md,
which the suite now reads. The operator half goes to the website.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 16:24:16 +05:00
Yan Bondarenko fe59a6bf91 fix(etcd): leave compaction to etcd and pass the real quota to defrag
The hourly defrag CronJob runs etcd-defrag with its defaults, which compact
to the latest revision before defragmenting. Every cluster already compacts
itself with a periodic five-minute retention, so the second compaction
either fails with "required revision has been compacted" when nothing was
written since, or drops the history that retention keeps for watchers to
resume from.

The same defaults evaluate dbQuotaUsage in the defrag rule against 2 GiB
rather than the quotaBackendBytes the cluster runs with, so the 80 percent
threshold fires at a different database size than intended.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-11 20:21:26 +03:00
Aleksei Sviridkin d9689bd6e7 fix(gateway): stop describing Cilium 1.19.5 as the shipped release
With Cilium at 1.20.2 the tenant gateway comments, README and one
status message still explained behaviour as that of "the pin" at
1.19.5. Several of those claims no longer hold: a Gateway with two
routed passthrough ports now gets an Envoy listener per port, Cilium
judges a route's kinds against the listener the route names, and the
shipped operator accepts an HTTPS and a passthrough listener on one
port when the passthrough hostname is strictly narrower.

Behaviour that is unchanged is re-anchored on the 1.20.2 source,
behaviour that is gone is stated as history, and the refusal of one
hostname on two passthrough ports no longer blames a Cilium limitation
the shipped release does not have.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin ad3f15d4f3 feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.

The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.

Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.

The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.

The backend resolution, its watches and the RBAC rules that served them
go away.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin 52f09d1aed fix(gateway)!: render passthrough services only where platform routes attach
The chart wrote the tlsPassthroughServices default onto every non-edge
tenant's TenantGateway, so every tenant Gateway carried tls-api,
tls-vm-exportproxy and tls-cdi-uploadproxy. The TLSRoutes that serve
those names are platform routes: the packages owning the services render
one per name in publishing.exposedServices while gateway.enabled is on,
parented to the publishing tenant's Gateway alone. Everywhere else no
shipped route attached to the listeners, and a passthrough listener
holds its name whether or not anything is routed through it.

The chart now writes the field only on the publishing tenant, with the
same gateway.enabled switch and the same exposedServices filter those
routes use, so a listener exists exactly where a platform route can
attach. A TLSRoute attached by hand to one of those listeners on another
tenant loses its parent.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 18:46:47 +03:00
Aleksei Sviridkin d624049a3a feat(gateway): allow tlsPassthroughListeners under dns01 and existingSecret
The refusal existed because Cilium 1.19.5 served every Gateway port
from one Envoy listener matched by SNI alone, so the "*.<apex>"
terminate listener of the wildcard modes and a passthrough listener
under the apex could reach each other's backend. The shipped Cilium
1.20.2 gives each port its own Envoy listener when SNI overlaps across
ports, and Gateway API treats listeners on different ports as
compatible, so the pair is served as declared. edge still refuses the
field: it renders no TLS listener for an entry to bind to.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:25 +03:00
Alexey Artamonov e63b9e439c fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.

The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.

The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 65f0caae2b fix(monitoring): keep a central tenant's write login out of its Grafana
A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.

Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov cc51cac63d fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one
login it writes with was also the login of the tenant's Grafana
datasource, and it could both read and write. vmauth cleans the request
path before it picks a route, so any user the tenant admits to its
Grafana, a Viewer included, could POST through the datasource proxy to
../insert and write into the shared store past the limiter.

Each central tenant now has two VMUsers on its one account: the
collector's, which may only write, and Grafana's, which may only read,
each with its own chart-owned Secret and password. A namespace name
holds no dot, so the reader's username never matches another tenant's.
On upgrade the datasource moves to the new login in the same release
that creates it, so reads may fail until vmauth loads the new user.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov fc0db49187 fix(monitoring): bound what a central tenant writes above its rate
Each collector pod starts with a full token bucket, and burst was at
least 64 MiB, the largest request vtinsert accepts. A tenant controls
when its collector pods start, through any Monitoring change or by
deleting them, and how many there are, so it could write tens of
mebibytes above its rate per pod start and gigabytes per rollout of
many replicas, which the two-day bound does not survive.

The limiter only sees requests the receiver admits, so in central mode
the receiver now caps them at 4 MiB, decompressed for HTTP, and burst
drops to that floor; a pod start lets at most 4 MiB through above the
rate. The replica count is capped at 10. The texts now say what the
default rate guarantees: one tenant writes at most a default store's
disk cap over two days counted uncompressed, rather than that it
cannot fill it, and they name the per-pod-start allowance and that a
sender pinned to one replica gets that replica's share.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 177b207a27 feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.

The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.

Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 32a6c71972 fix(monitoring): keep the collector image out of a tenant's reach
The apps API defaults a Monitoring spec from its schema but does not
prune keys the schema leaves out, and the Monitoring chart hands its
values to the inner chart as they are. A tenant could therefore set
tracingCollector.image and run any collector it liked in its
namespace; with a per-tenant rate limit in that collector, the same
key would let a tenant drop the limit. The image is the platform's,
so it is no longer passed through.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Aleksei Sviridkin bf03c84711 chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does

This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.

The four commits are meant to be read one by one, and each builds and
passes the tests on its own:

1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.

Commits 2 and 4 are where the review effort goes; 1 and 3 are small.

Known changes in behaviour:

- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.

The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 21:45:51 +03:00
Aleksei Sviridkin 64f43f4f77 fix(computeplane): do not grant tenants the cluster admin kubeconfig
The computeplane module deploys the kubernetes chart into the tenant
namespace, and that chart's dashboard Role lets the tenant's "use"
subjects read <release>-admin-kubeconfig. Withholding the Secret from
the tenant secrets view does not help when RBAC grants it directly, so
any tenant user at "use" level or above could fetch the cluster-admin
credential of a cluster the module promises to keep operator-owned.

The kubernetes chart now drops that rule when the internal value
withholdAdminKubeconfig is set, and the module sets it in the cluster
HelmRelease's inline values. The tenant cannot reach those values: the
module reads only nodeGroups from them and splices an allowlisted
subset into the pool releases, and Flux applies spec.values over
valuesFrom. A regular kubernetes app keeps the rule, so its dashboard
can still offer the kubeconfig.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:36:57 +03:00
Alexey Artamonov 68ee97300b docs(monitoring): say a central tenant's collector still needs enabling
The tracingStorages description said a tenant with shared-central
tracing gets the collector without a local entry, which reads as if
tracingCentral alone deploys it. tracingCollector.enabled still
decides whether it renders.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-01 18:02:22 +03:00
Alexey Artamonov d5fc723208 feat(monitoring): bound every traces store by disk usage
The distributed tracing design requires each traces backend to be
disk-bounded, not only age-bounded, so a full volume cannot block
ingest. retentionDiskUsageBytes was optional, so a store without it
filled its volume and stopped taking spans; with a shared-central store
that stops ingest for every tenant at once.

A store without a byte cap now drops its oldest day once its
filesystem is 80% full. VictoriaTraces refuses to start with both
limits set, so a byte cap, when given, replaces the percentage one.
It always keeps the last two days, which the values text now says.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 23:29:41 +03:00
Alexey Artamonov 27b7376bb9 feat(monitoring): add a shared-central tracing topology
A per-tenant traces backend costs every tenant its own VictoriaTraces
stack. A tenant can now send new spans to one shared store in
tenant-root instead, switched on by _namespace.tracingCentral, which
the tenant chart publishes and the apps API does not let an
application set. tenant-root decides whether it hosts that store, with
tracingCentralHost next to a cluster-mode store named generic.

VictoriaTraces takes the tenant account from request headers and does
no authorization of its own, so other tenants reach the store only
through a vmauth in tenant-root; tenant-root itself, with whoever it
admits to its Grafana, stays inside the trust boundary. Each central
tenant gets a VMUser that pins an account derived from its namespace
name and UID for both writes and reads and caps its requests in
flight, which the collector's two export senders stay under; its
collector exports and its Grafana reads with credentials the chart
generates and keeps, and vmauth overrides any account a client sends.
The chart owns those credentials because the operator publishes a
VMUser's own Secret only once a VMAuth selects it; the collector rolls
when they change, since it reads them at start. The vmauth internal
routes, whose /metrics names every tenant's user, are served on a port
tenants cannot reach, and the vmauth is sized by a VPA, spread across
nodes under a disruption budget, has a WorkloadMonitor like the other
monitoring workloads, and may reach the apiserver, where its
config-reloader reads its config Secret.

Renders refuse the states in which spans would be dropped with every
object Ready: a central tenant while tenant-root hosts nothing,
tenant-root dropping the hosting while central tenants remain, and
the host flag without a store to host. Each refusal reads the other
side at render time, so a concurrent opt-in and stop-hosting can pass
both; the values text names that race and its recovery. At run time
tenant-root's release is not Ready while the vmauth has failed.
Only the collector's export moves: local stores the tenant still lists
keep rendering with their datasources, because the switch belongs to
the parent tenant and the stores to this one, and the one-store limit
of the collector holds in both modes, since turning central off makes
it export locally again. tenant-root never counts as central.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-30 22:23:47 +03:00
Yan Bondarenko 17aaf3e038 feat(monitoring): add optional email alert delivery
Platform alerts need an email delivery path alongside Alerta without
exposing arbitrary namespace Secrets through tenant-controlled values.
SMTP credentials therefore use a fixed Secret name.

Alertmanager parses the addresses and connects to the SMTP host only
when it sends a notification, so a malformed value there drops alerts
instead of failing the release. The chart validates these fields when
it renders. Addresses must be literal dot-atom mailboxes, and a domain
literal must hold a dotted-quad IPv4 address, because IPv6 literal
syntax differs across the supported Go mail parsers. A bracketed
smarthost must be a valid IPv4 or IPv6 address, and its port must be
between 1 and 65535 without leading zeros. The repeat interval follows
the Alertmanager duration grammar and range, so a value that renders is
one Alertmanager loads.

Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
2026-09-30 13:55:12 +03:00
Mattia Eleuteri 7c9ab0e28e fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU
cozy-lib.resources.sanitize divides the CPU limit by cpuAllocationRatio
and rendered the quotient as a bare float. When the ratio does not
divide the value evenly the result is finer than a milli-CPU, and the
VPA admission webhook rejects it outright:

  admission webhook "vpa.k8s.io" denied the request:
  minAllowed: CPU [62500u] must be a whole number of milli CPUs

packages/extra/etcd routes its VPA minAllowed/maxAllowed through the
helper, so 250m / 4 = 62.5m left the etcd HelmRelease looping
UpgradeFailed -> RollbackFailed forever: the rollback target renders the
same invalid value. Since v1.6.0 HelmReleases wait on each other, so
that one stuck app also blocks tenant-root, cozystack-basics and
kubevirt-cdi-operator, and without CDI the Talos worker rollover cannot
import the worker OS image. Only the divisors of 250 worked as ratios,
which is an artefact of one chart's hardcoded floor rather than a real
constraint on the operator.

The request is rounded up, never down, so it stays at or above the
intended fraction of the limit. It is rounded only when it has to be:
if the milli-CPU value equals its own ceiling, the float keeps its
current rendering, so nothing changes on the default ratio of 10 or on
any ratio that divides evenly; otherwise it renders as the ceiling in
m. The comparison is exact, with no tolerance. sprig's divf and mulf run
on shopspring/decimal rather than float64, so mulf (divf 0.25 10.0)
1000.0 renders exactly 25 and there is no binary-float noise on this
codepath to absorb. A tolerance would reintroduce the bug: a value
whole-milli only to within it, such as 100.0000004m at ratio 1, would
read as exact and render "0.1000000004", rejected for the same reason
62.5m is. The same predicate covers sub-milli input, since ceil carries
any positive request to at least 1m, and a zero request stays "0".

Fixes #2917

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:04:00 +02:00
Aleksei Sviridkin 02461acdde Merge remote-tracking branch 'origin/main' into feat/arm64-bundle
Signed-off-by: Aleksei Sviridkin <f@lex.la>

# Conflicts:
#	hack/image-builders-cross-compile.bats
2026-09-28 21:42:18 +03:00
IvanHunters 2fa51bf587 test(charts): isolate guard coverage for every guarded template
Add per-template guard suites so each guarded hostname site is exercised
on its own: harbor harbor.yaml and httproute.yaml, bootbox machines.yaml,
seaweedfs cosi-deployment.yaml, bucket deployment.yaml and httproute.yaml,
monitoring alerta.yaml, grafana httproute.yaml and the grafanaHost helper
via oidc-keycloak.yaml. Each suite renders only its target template, so
removing that file's guard reddens it. Also add a postgres positive case
that renders the Certificate with an apex set (external=true).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 18:32:49 +03:00