Commit Graph
8093 Commits
Author SHA1 Message Date
Aleksei Sviridkin e4d059cedd fix(etcd): require an even spread of members over nodes by default (#4986)
## What this PR does

The tenant etcd chart spreads members with `maxSkew: 1` on
`kubernetes.io/hostname` and `whenUnsatisfiable: ScheduleAnyway`, the
default since #119. `ScheduleAnyway` only adds to the score of a node
and other scores can outweigh it, so two members of one etcd can still
start on the same node, where a node-local volume then keeps them. This
PR makes the default `DoNotSchedule` with `nodeTaintsPolicy: Ignore`.
The branch for `_cluster.scheduling.globalAppTopologySpreadConstraints`
is untouched, so a platform that sets its own policy renders exactly as
before, and the soft pod anti-affinity from #3962 stays.

Seen on a three-node cluster running v1.6.4, whose `etcd-cluster.yaml`
and etcd-operator v0.5.5 are the same as in v1.6.5. A new tenant etcd
got the Pods of its first two members on one node, created 26 seconds
apart, and the third on another node 18 seconds later. The Pods carry
the `ScheduleAnyway` constraint and were scheduled by
`linstor-scheduler`, whose profile adds the LINSTOR extender's
prioritize score to the same sum with weight 5. The data volumes are on
a node-local StorageClass with `WaitForFirstConsumer` and
`allowRemoteVolumeAccess: "false"`, so the two members cannot leave that
node, and losing it takes the quorum. A second tenant etcd created there
the next day came up the same way. On a test cluster four tenant etcds
created within one week each had two members on one node, and we moved
one member of each by hand.

`DoNotSchedule` does not stop an etcd with more members than nodes from
starting when every node can run its Pods, unlike a required
anti-affinity, because the skew is counted between the nodes that exist.
kube-scheduler rejects a node only when the matching Pods on it plus the
new one exceed the node with the fewest by more than `maxSkew`
([filtering.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/filtering.go#L334-L352)).
One node takes all three members, two nodes take two and one, and three
or more nodes take one each. A member whose only allowed node cannot
take it, for example because the storage pool there is full, now waits
Pending instead of joining another member.

`nodeTaintsPolicy: Ignore` is the Kubernetes default and is written out
on purpose. A cordoned node carries `node.kubernetes.io/unschedulable`,
and `Honor` would take it out of the count
([common.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/common.go#L50-L54)),
so a member created during a drain would start next to another member
and stay there on a node-local volume. With `Ignore` it waits Pending
until the node is back, while the other two keep quorum. The cost is
that a node no member can run on, such as a tainted control plane,
counts as an empty node too, so on a cluster with such a node each
schedulable node takes at most one member and the members beyond that do
not start. #4947 makes the same choice for the SeaweedFS masters and
checked both policies in kwok. A platform that wants the old behaviour
can set `scheduling.globalAppTopologySpreadConstraints` to
`ScheduleAnyway`, which the other charts that read it pick up as well,
and a new test case pins that the chart renders such a policy as
written.

On upgrade nothing is recreated. The etcd-operator adopts a changed
`spec.topologySpreadConstraints` into `status.observed` once the cluster
has reached its current target, or its progress deadline has passed
([etcdcluster_controller.go#L203-L214](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L203-L214)).
It copies the value from there into the spec of an `EtcdMember` only
when it creates one, at bootstrap
([#L446](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L446))
and on scale-up
([#L851](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L851)),
and nothing in the operator writes it into an existing `EtcdMember`. The
member controller creates a Pod only when it is missing and builds it
from that member's own copy
([etcdmember_controller.go#L499-L512](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L499-L512),
[#L545-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L545-L549),
[#L860](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L860)).
So running members keep `ScheduleAnyway`, and so does a member whose Pod
is evicted, deleted or replaced after a terminal phase
([#L164-L186](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L164-L186)),
since the new Pod is built from the same copy. Only members created
after the upgrade get `DoNotSchedule`, which means a scale-up, a
crash-loop replacement or a member whose `EtcdMember` was deleted. No
member is rolled, neither one at a time nor all at once. This matches
the field docs
([etcdcluster_types.go#L542-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdcluster_types.go#L542-L549),
[etcdmember_types.go#L112-L116](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdmember_types.go#L112-L116)).
The code paths are the same in v0.5.8, and in v0.5.5, which v1.6.4 and
v1.6.5 ship, except that v0.5.5 does not yet replace a Pod left in a
terminal phase.

Members that already share a node therefore stay there. One of them
moves only when a new `EtcdMember` takes its place, and the constraint
then sends the replacement to the node without a member. That is how we
moved the members on the test cluster, with `DoNotSchedule` set on the
`EtcdCluster` by hand. The operator's docs say a scheduling change can
be applied by deleting one Pod at a time
([concepts.md#L272](https://github.com/cozystack/etcd-operator/blob/v0.5.6/docs/concepts.md#L272)),
but the recreated Pod gets the member's old copy, so that does not move
it.

The release note now declares two cases from review. In the first, an
etcd created while the cluster had fewer nodes than members has two or
three members on one node, and once a node is added a recreated Pod of
one of them stays Pending, since `maxSkew` rejects its own node and its
node-local volume rules out the others. Deleting that member's
`EtcdMember` while the other members keep quorum replaces it on a node
without a member. In v0.5.5 the member finalizer removes the member from
etcd through the remaining members before the object goes
([`handleDeletion`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L195-L231),
[`removeMemberFromEtcd`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L243-L310)),
its Pod and its `data-<member>` PVC are owned by it and deleted with it
([#L403](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L403),
[#L490](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L490)),
and the cluster controller waits for the deletion, then adds a member
under a new name with the constraints from `status.observed`
([etcdcluster_controller.go#L254-L312](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L254-L312),
[`scaleUp`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L820-L851)).
Its new PVC binds where its Pod lands, and under `DoNotSchedule` only a
node without a member passes. The operator documents the same steps for
a broken member, along with the limit that `MemberRemove` cannot run
without quorum
([operations.md#L208-L231](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L208-L231)),
and v0.5.6, which main and `release-1.6` ship now, has the same path.
The upstream chart of the operator installs a policy that denies
`EtcdMember` deletion without the
`etcd-operator.cozystack.io/allow-deletion=true` annotation
([member-deletion-policy.yaml](https://github.com/cozystack/etcd-operator/blob/v0.5.5/charts/etcd-operator/templates/member-deletion-policy.yaml)),
and the `etcd-operator` package here does not install it.

On a three-node cluster running v0.5.5 a tenant etcd had two members on
one node with `DoNotSchedule` in effect. We deleted the `EtcdMember` of
one of the two while the other members kept quorum. The replacement's
Pod and its new node-local PVC were scheduled on the node without a
member, the member joined, and the `EtcdCluster` stayed `Available` with
quorum the whole time, polled every 5 seconds.

In the second case `nodeTaintsPolicy: Ignore` counts a node the Pods
cannot tolerate as one with zero members, so on a cluster with such a
node and fewer schedulable nodes than members the members beyond the
schedulable nodes do not start. The way out is the soft global policy in
the platform `Package`, set before the upgrade.

```yaml
apiVersion: cozystack.io/v1alpha1
kind: Package
metadata:
  name: cozystack.cozystack-platform
spec:
  components:
    platform:
      values:
        scheduling:
          globalAppTopologySpreadConstraints: |
            topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: ScheduleAnyway
```

The platform chart writes `scheduling` into `_cluster` of the
`cozystack-values` Secret
([apps.yaml#L292-L295](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/core/platform/templates/apps.yaml#L292-L295)),
the tenant chart copies `_cluster` into the Secret of each tenant
namespace
([namespace.yaml#L109-L120](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/apps/tenant/templates/namespace.yaml#L109-L120)),
and this chart renders the value as written, which the new test case
pins. Rendering the platform and tenant charts with the value above, on
main and on `release-1.6`, gives the same string in both Secrets. The
operator adopts a changed spec into `status.observed` only once all
members are Ready or the progress deadline has passed
([etcdcluster_controller.go#L196-L213](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L196-L213)),
and it clears the deadline when the cluster converges
([#L1448-L1456](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L1448-L1456)).
A new etcd stuck short of its members adopts the policy once its first
deadline passes, and deleting the `EtcdMember` of the Pending member
then brings a replacement that starts. An etcd that had converged before
one of its replacements went Pending adopts the policy only after the
deadline is forced as the operator docs describe
([operations.md#L197-L206](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L197-L206)),
which is why the release note says to set it before the upgrade.

#4817 moves this block into `_etcd.tpl` in cozy-lib without changing it,
so whichever of the two lands second has to carry the two lines over.

`tests/topology-spread_test.yaml` now pins the whole default constraint,
and a new case pins that a `ScheduleAnyway` policy from
`globalAppTopologySpreadConstraints` renders as written, without
`nodeTaintsPolicy`. The existing zone override case is unchanged.
Against the template on main the suite fails only the default case, 1
failed and 25 passed. Switching the default to `Honor`, to
`ScheduleAnyway` or to `maxSkew: 2` fails the default case, and adding
`nodeTaintsPolicy` to the override branch fails the new case.

```console
$ docker run --rm -v "$PWD":/apps -w /apps/packages/extra/etcd helmunittest/helm-unittest:4.2.3-1.1.2 .
Charts:      1 passed, 1 total
Test Suites: 9 passed, 9 total
Tests:       26 passed, 26 total
```

On main the same command gives 25 tests in 9 suites, with helm-unittest
1.1.2 as CI pins it. With default values `helm template` differs from
main in the two lines and in the comment above the affinity block, and
with a global policy set only in that comment. The docs commit changes
the `affinity` description, from which `make generate` rewrites the
README, `values.schema.json` and the `ApplicationDefinition` with
nothing but that text, and the suite stays at 26 tests. I have not
rolled this chart change out on a cluster.

The spread block and the test file are the same on release-1.6 and
release-1.7. On release-1.6 the template has no affinity block yet, so
the fix commit cherry-picks without conflicts and the docs commit has
nothing to change there. That backport is a separate PR.

### Screenshots

Not a UI change.

### Downstream repositories

One changed and one added line in
`packages/extra/etcd/templates/etcd-cluster.yaml` and its test suite,
and new text for the `affinity` description and the comment above the
affinity block. The description reaches `values.schema.json`, the README
and the `ApplicationDefinition` as text only, with no key, type or
default change, and the website rebuilds the etcd reference page from
the README on release. The website mentions
`globalAppTopologySpreadConstraints` only as a platform setting, which
stays as it is, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(etcd): tenant etcd members now spread over nodes with whenUnsatisfiable DoNotSchedule by default, so two members of one etcd no longer start on the same node while another node has none. Members that exist are not moved, only members created after the upgrade get the new constraint.

An etcd created while the cluster had fewer nodes than members puts two or three members on one node. Once a node is added, such a member whose Pod is recreated (drain, eviction, graceful node shutdown) stays Pending, because maxSkew rejects its own node and its node-local volume rules out the others. With two nodes grown to three the etcd runs two of three members and its PodDisruptionBudget blocks draining the other nodes. With one node grown to three a reboot of that node brings back one member of three and quorum is lost. A stranded member is replaced by deleting its EtcdMember while the other members keep quorum. Its Pod and PVC go with it, and the replacement gets a new Pod and PVC on the node without a member. Right after adding nodes, doing this for one co-located member at a time while all members are Ready avoids the case.

With nodeTaintsPolicy Ignore a node the etcd Pods cannot tolerate, such as a tainted control plane, counts as a node with zero members, so no node can take a second member. On a cluster with such a node and fewer schedulable nodes than members, a new etcd starts only as many members as there are schedulable nodes, one of three on a single schedulable node. An existing etcd there keeps its members, but a member it replaces stays Pending the same way.

Either case is avoided by setting scheduling.globalAppTopologySpreadConstraints in spec.components.platform.values of the cozystack.cozystack-platform Package to a topologySpreadConstraints entry with maxSkew 1, topologyKey kubernetes.io/hostname and whenUnsatisfiable ScheduleAnyway before the upgrade. Every app chart that reads this value applies it, two members of a new etcd can share a node again, and the operator gives it only to members created after it adopts the changed EtcdCluster spec into status.observed. Once status.observed.topologySpreadConstraints of an EtcdCluster shows ScheduleAnyway, a member of it that stays Pending is replaced by deleting its EtcdMember.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **Bug Fixes**
- Default etcd placement now requires even spreading across hosts and
ignores node taints when evaluating placement.
- Configurable cluster-wide spreading continues to use a flexible
scheduling policy when selected.
- **Documentation**
- Clarified how affinity and topology spreading work together, when
required anti-affinity or node-local storage can leave members pending,
and that affinity changes take effect as members are replaced.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:39:53 +03:00
Aleksei Sviridkin 76204b10aa fix(ingress-nginx): raise client keepalive_time above the 1h default (#5250)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

ingress-nginx exposes no ConfigMap key for the client-side
`keepalive_time`, so nginx keeps its built-in 1h default: every client
connection older than an hour is closed, and on HTTP/2 that is a
`GOAWAY` (`ErrCode=NO_ERROR`). A client that multiplexes a long
operation over one connection loses the request in flight. We hit it
with a kopia repository maintenance run by Velero against a SeaweedFS S3
endpoint published through this controller: the run opened its
connection at 18:57:05, finished the heavy part, and failed at 19:57:38
while listing blobs with `http2: server sent GOAWAY and closed the
connection`, because kopia does not retry that listing. A shorter
maintenance on the previous day succeeded on the same repository.

This sets `keepalive_time 12h` through `http-snippet` in the controller
ConfigMap. Idle connections are still closed after `keep-alive` (75s),
so only connections that are actually in use live longer. No cozystack
template sets `http-snippet` today; the tenant ingress HelmRelease only
sets other `controller.config` keys, which Helm merges with this
default, and a test covers that merge.

Applied by hand on a production cluster before this PR: the controllers
reloaded cleanly and S3 traffic was unaffected.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(ingress-nginx): raise the client keepalive_time from the nginx 1h default to 12h, so long-lived HTTP/2 connections (for example kopia maintenance against S3) are no longer cut by a GOAWAY after one hour
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* The ingress controller now configures NGINX to allow HTTP connections
to remain open for up to 12 hours, supporting long-running operations
such as S3 transfers.
* **Tests**
* Added checks confirming the setting remains in place when PROXY
protocol and whitelist configuration are enabled.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:32:52 +03:00
Aleksei Sviridkin 7ec746a360 fix(monitoring): select the vmalert Service the operator creates (#5094)
## What this PR does

`templates/vm/vmalert-scrape.yaml` selects `app.kubernetes.io/instance:
vmalert`. The chart names the VMAlert `vmalert-<first metricsStorages
name>` (`templates/vm/vmalert.yaml` renders only the first storage), and
the operator labels its Service with that name, so with the default
storages the Service `vmalert-vmalert-shortterm` carries
`app.kubernetes.io/instance=vmalert-shortterm`. The VMServiceScrape
matches no Service and the metrics of vmalert itself never reach the
storage.

Seen on three v1.6.5 clusters. The `vmalert` VMServiceScrape in
`tenant-root` selects no Service, `count({__name__=~"vmalert_.*"})` in
vmselect-shortterm returns nothing while
`count(up{job="vmalertmanager-alertmanager"} == 1)` returns 3, and the
`/metrics` endpoint of vmalert does serve those series. No other scrape
object covers it, and no rule under `packages/` uses `vmalert_*`
metrics, so rule evaluation and notifier errors of vmalert are not
visible anywhere in the storage.

The scrape dates from v0.0.1, when the VMAlert was called `vmalert`.
4471b4ba (#289) renamed it to `vmalert-<storage>` in August 2024 and the
selector stayed, so the metrics of vmalert have not been collected
since.

The selector now takes the instance from the first `metricsStorages`
entry, the one `vmalert.yaml` renders, and the scrape is not rendered
when there are no storages, like the VMAlert itself. Keeping `instance`
instead of dropping it keeps the selector pinned to the one VMAlert this
release owns.

A new suite `tests/vmalert_scrape_test.yaml` covers the selector with
two storages and the empty case. `helm unittest .` in
`packages/system/monitoring` passes 306 of 306 with helm-unittest 1.1.2
as CI pins it, and against the template on main both new cases fail. A
full `helm template` of the chart with the default storages renders the
scrape with `app.kubernetes.io/instance: vmalert-shortterm`. I have not
run the change on a live cluster.

The template is the same on release-1.6, and that backport is a separate
PR.

### Screenshots

Not a UI change.

### Downstream repositories

One chart template and a new unittest suite in
`packages/system/monitoring`. No values key, schema, default, README or
package change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring): vmalert metrics are collected again. The vmalert VMServiceScrape selected app.kubernetes.io/instance=vmalert while the Service is labelled vmalert-<first metrics storage>, so it matched nothing.
```
2026-11-13 15:16:23 +03:00
Aleksei Sviridkin 900baf2215 fix(linstor): update piraeus-operator to v2.12.1 (#5166)
## What this PR does

Updates the LINSTOR stack to the current stable piraeus-operator
release: operator v2.12.1, LINSTOR server 1.35.3 and LINSTOR CSI
v1.13.2, the last two built here with our patches. This started as a fix
for the NFS server image hanging up the host console on bare-metal
storage nodes (Ubuntu 24.04, Cozystack 1.6.5 and 1.7.0). That needed
nfs-server v1.10.6 or later. Operators also asked for per-volume IO
limits, which need linstor-csi v1.11.3 or later and piraeus-operator
v2.10.8 or later, so the update goes to the current release rather than
the minimum.

The first commit fixes a problem the update exposed. The operator runs
the LINSTOR database migration in a `run-migration` init container, and
this package pinned only the main controller container to our build.
With the operator update, that init container ran the operator's default
LINSTOR 1.35.3 and migrated the database before the controller image
changed. The 1.33.3 controller then crash-looped until the linstor
release caught up. Both init containers that run the LINSTOR server
image (`run-migration` and the satellite's `setup-lvm-configuration`)
now use the platform image, with a test pinning it. This protects later
upgrades where the operator default is newer than our build. It does not
help this upgrade: the pin ships in the linstor release, and Flux
reconciles piraeus-operator first.

Patches:

- linstor-csi: the Protocol C override for dual attach is dropped. Since
v1.12 the driver asks LINSTOR to manage dual-primary, and LINSTOR 1.35
sets protocol C on the connections that need it. Upstream closed the
same change as superseded (piraeusdatastore/linstor-csi#435). The
relocate-after-clone patch is carried forward.
- LINSTOR server: six of the nine patches are in upstream by 1.35.3 and
are removed. `fix-k8s-create-already-exists`,
`retry-adjust-after-stale-bitmap` and `native-protoc` are carried
forward. LINSTOR 1.34 requires Java 21, so the image moves from bookworm
to trixie.

Behaviour that changes for users:

- No downgrade. LINSTOR 1.35 migrates the database and drops the `SEC_*`
tables, so a cluster that has run this version cannot go back to 1.33.
- The LINSTOR controller is down for a short time during this upgrade.
piraeus-operator is reconciled before the linstor release, so the
database is migrated to 1.35 while the old controller still runs. That
controller crash-loops until the linstor release rolls out, and if the
rollout fails there is no way back.
- A volume on a Protocol A or B resource group that was attached to two
nodes while the CSI driver was updated (a live migration in progress)
keeps the Protocol C override the old driver set on its resource
definition. The new driver does not remove it. Data is not affected, the
resource just stays on protocol C. Such resource definitions have the
`Aux/csi-protocol-override` property. To remove the override, run
`linstor resource-definition set-property <rd> DrbdOptions/Net/protocol`
and `linstor resource-definition set-property --aux <rd>
csi-protocol-override`.
- The CSIDriver moves to `fsGroupPolicy: File`, so `fsGroup` is also
applied to ReadWriteMany filesystem volumes.
- The operator manages Secrets only in its own namespace, and the CSI
controller loses cluster-wide read access to Secrets. Nothing in this
repository references snapshotter Secrets.
- The csi-health-monitor sidecar is removed.
- LINSTOR's default evict timeout goes from 1h to 12h; this package does
not set it.
- Volume group snapshots turn on only when
`groupsnapshot.storage.k8s.io/v1` is served. The snapshot controller
here still serves v1alpha1, so they stay off; #5239 changes that.
- IO limits can be set through `qos.linbit.com/{rbps,wbps,riops,wiops}`
StorageClass parameters. Enforcing them needs NRI enabled on the nodes
and the nri-volume-qos plugin, neither of which this PR adds.

Tested on a three-node Talos cluster, upgrading from the current main
packages to this branch:

- LINSTOR database migrated to 1.35.3, no faulty resources, no error
reports.
- Data written before the upgrade is intact, checked by checksum.
- A snapshot taken before the upgrade restores, and all three replicas
come up `UpToDate`; cloning and new snapshots work.
- KubeVirt live migration works for a ReplicatedBlock volume, including
one on a Protocol A resource group: LINSTOR switches the connection to
protocol C for the migration and restores A afterwards.
- A StorageClass with QoS parameters provisions, and the limits reach
the resource definition and the VolumeAttachment metadata.
- ZFS userspace in the satellite is now 2.4.4, matching the kernel
module.

Not tested: an arm64 build.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): update piraeus-operator to v2.12.1, LINSTOR to 1.35.3 and LINSTOR CSI to v1.13.2. Fixes the NFS server breaking console login on storage nodes and adds IO QoS StorageClass parameters. LINSTOR 1.35 migrates its database, and a downgrade to an earlier version is not possible after the upgrade.
```
2026-11-13 15:15:43 +03:00
Aleksei Sviridkin 12393ec3d3 fix(etcd): size the defrag job for the client it runs (#5100)
## What this PR does

The hourly `etcd-defrag` CronJob runs the `ghcr.io/ahrtr/etcd-defrag`
client, which only sends defragment calls to the members, but it
requested 200m CPU and 256Mi from the tenant quota and limited itself at
500m and 512Mi. On a v1.6.5 sandbox tenant created through the panel,
its own etcd and SeaweedFS took 1525m of the 1600m CPU quota, every
hourly job failed with `exceeded quota` before a pod was created, and
the defrag never ran once. Over a day on another v1.6.5 cluster the
client peaked at 5.1 MiB working set, and its runs are too short to show
up in a CPU rate.

The container now requests 10m and 32Mi and is limited at 100m and 64Mi,
so it fits a tight tenant quota and still leaves the client room above
what it was seen to use. The tenant LimitRange only sets defaults, so
explicit small values are not rejected.

A new case in `tests/defrag_test.yaml` pins the resources. `helm
unittest .` in `packages/extra/etcd` passes 26 of 26 with helm-unittest
1.1.2, and against the template on main the new case fails and the other
25 pass. I have not run the change on a live cluster.

The template is the same on release-1.6, and that backport is a separate
PR.

### Screenshots

Not a UI change.

### Downstream repositories

One template of `packages/extra/etcd` and its test. No values key,
schema, default or package change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(etcd): the tenant etcd defrag job requests 10m and 32Mi instead of 200m and 256Mi, so it runs in tenants whose quota is nearly used up by their own services.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Updates**
* The etcd defragmentation job now has lower CPU and memory requests and
limits: requests are 10m CPU and 32Mi memory, and limits are 100m CPU
and 64Mi memory. This reduces the resources configured for the job.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:14:07 +03:00
Aleksei Sviridkin 7c46e1e384 fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does

The barman-cloud sidecar of Postgres backups is deliberately left
without a CPU limit, so that a throttled sidecar slows WAL archiving
down instead of failing it. In a tenant with `resourceQuotas` set that
intent does not hold: `packages/apps/tenant/templates/quota.yaml`
creates the `tenant-range-limits` LimitRange, whose default CPU limit is
`250m`, and the sidecar receives it. Full backups and WAL segments are
then gzip-compressed on a quarter of a core. On a production tenant the
first full backup of a 1.6 GB database took 30 minutes, uploading at
about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small
database on the same destination finished in 35 seconds.

This sets an explicit `limits.cpu: "1"` on the sidecar, in the two
places that build its configuration: `barmanSidecarConfiguration` in the
backup driver (platform-managed ObjectStores) and
`cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the
postgres and keycloak charts). This mirrors how the memory request and
limit there already override the LimitRange's 128Mi default. One core
removes the 250m ceiling without letting the sidecar starve the
instance. Inside a tenant it counts against the `limits.cpu` quota, one
core per instance pod; the comments now say so.

The sidecar is injected when the instance pod is created, so existing
instances pick the new limit up at their next restart.

Tests: the Go test that asserted the absence of a CPU limit now asserts
one core, and so does
`packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail
with the implementation reverted and pass with it. `go test
./internal/backupcontroller/`, `helm unittest packages/apps/postgres`
(220) and `helm unittest packages/system/keycloak` (121) pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (the backup driver, `cozy-lib`'s
barman template, two tests): no app, value, schema, default, CRD,
annotation or tooling change, so none of the listed repositories
restates anything touched here.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart.
```
2026-11-13 14:54:51 +03:00
Aleksei Sviridkin 30c4232be1 fix(seaweedfs): stop the master crashing on the crowded volume map (#5308)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map
read/write on VolumeLayout.crowded", released in 4.46) as a patch on the
in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`.

In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map
under `accessLock.RLock()`, while
`(*VolumeLayout).GetWritableVolumeCount` reads the same map under
`RLock()` on the `Assign` path. Two read-lock holders, one writing and
one reading the map, make the Go runtime abort the master with `fatal
error: concurrent map read and map write`. This cannot be recovered, so
the process exits and S3 requests that need an assign fail until a
master is back. The fix takes the write lock in `SetVolumeCrowded`;
every other mutation of `crowded` already holds it.

`patches/volume-layout-crowded-lock.diff` is the upstream squash commit
5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`),
regenerated against the 4.45 source so it applies without offset. Its
content is unchanged, including the regression test
`TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the
Dockerfile's existing `go test` step runs on every build.

#### Seen in production

On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS
4.45), with a dedicated S3 cluster used as the Velero backup target:

- Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every
night with exit code 2 under the backup load, between 01:00 and 03:00
UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts
per master over 8 days.
- The stack is always the same:
  ```
  fatal error: concurrent map read and map write
  ...

github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...)
      weed/topology/volume_layout.go:834
  github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...)
      weed/server/master_grpc_server_assign.go:97
  ```
- Clients see HTTP 500 `InternalError` from the S3 gateway, and the
nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3
`PartiallyFailed` on the night of 2026-11-12 to 13).

Upstream issue: seaweedfs/seaweedfs#11211.

#### Testing

- `git apply --check` of both patches against the 4.45 source archive
(`SOURCE_SHA256` in the Dockerfile matches).
- Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as
in the image build: `fatal error: concurrent map writes`, FAIL. With the
fix: `go test ./weed/topology/` passes, and the new test passes under
`-race` too.
- `docker buildx build images/seaweedfs --platform linux/amd64` builds,
running the patched packages' tests.
- `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the
naming-guard check and the image/chart version check pass.

#### Lifecycle

The patch is dropped once the vendored chart reaches 4.46 or later, for
example with #4879 (4.48 bump). From that release on it no longer
applies, and `make update-image` fails until it is removed, as the
Dockerfile header says. This PR is kept small so that it does not wait
for #4879.

#### Backport

This is a `fix` with no CRD, migration or API change, and `release-1.7`
carries the same `images/seaweedfs/` as `main`, so the backport batch
should take it onto `release-1.7` with a clean cherry-pick for the next
v1.7.x. Clusters on v1.7 are the ones that crash today.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a
Go patch and a Dockerfile comment): no package added or renamed, no
values, schema, CRD, `hack/` or installer change, so no entry of the
trigger map applies.

### Release note

```release-note
fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved reliability when updating crowded-volume state concurrently
with writable-volume queries.
* **Documentation**
* Added notes about the upstream fix and when the local patch can be
removed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 14:52:45 +03:00
Aleksei Sviridkin 3fce040268 Merge branch 'main' into fix/etcd-strict-member-spread
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:49 +03:00
Aleksei Sviridkin d1f95e3212 Merge branch 'main' into fix/etcd-defrag-small-requests
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:32 +03:00
Aleksei Sviridkin 4f19e038c9 Merge branch 'main' into fix/monitoring-vmalert-scrape
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:18 +03:00
Aleksei Sviridkin 5e07258acb Merge branch 'main' into feat/ingress-nginx-keepalive-time
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 12:44:57 +03:00
Aleksei Sviridkin 56224a156c feat(virtual-machine): let each disk set its own boot order (#5266)
## What this PR does

A `vm-instance` boots from its disks in the order they appear in
`disks`: the template hardcodes `bootOrder: {{ add $i 1 }}`. Booting
from installer or rescue media, such as an ISO attached as a CD-ROM next
to the system disk, therefore means moving that media to the front of
the list, and anything that treats `disks[0]` as the system disk then
finds the ISO there instead.

This adds an optional `bootOrder` to each entry of `disks`:

- When set, it is the disk's boot priority (1 is tried first),
independent of its position in the list.
- When unset, the disk keeps its positional order (`index + 1`), so
existing VMs render exactly as before.
- The values schema requires `bootOrder >= 1`.
- Duplicate effective boot orders are refused at render time with the
names of both disks, rather than surfacing as a KubeVirt admission error
on the VirtualMachine. This includes an explicit value colliding with
another disk's positional one, so booting from a CD-ROM in second
position means setting both disks:

```yaml
disks:
  - name: system
    bootOrder: 2
  - name: install-iso
    bus: sata
    bootOrder: 1
```

`make generate` was run in the package, so the README,
`values.schema.json`, the Go type in `api/apps/v1alpha1/vminstance`
(with its regenerated deepcopy, since the field is a pointer) and the
`vm-instance-rd` OpenAPI schema are all updated.

A new suite, `tests/disk_boot_order_test.yaml`, mocks the DataVolumes
the template looks up and covers: positional order when no disk sets
`bootOrder`; a CD-ROM in second position booting first while the list
order and volumes stay unchanged; an explicit value colliding with a
positional one; two equal explicit values; and a value below 1 rejected
by the schema. Run against the previous template, the CD-ROM and
collision cases fail. `helm unittest packages/apps/vm-instance`: 6
suites, 77 tests passing.

### Screenshots

No UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

This adds a field to `vm-instance`'s `values.schema.json`, which the
trigger map routes to `cozystack/terraform-provider-cozystack`: its
`vminstance_schema.go` and `vminstance_model.go` model `disks` by hand,
so `bootOrder` needs adding there. The follow-up has not been opened yet
and the box is left blank until it is linked. The website reference page
is regenerated from the README by the release bot; nothing else in the
map is touched.

### Release note

```release-note
feat(virtual-machine): disks in a vm-instance accept an optional `bootOrder`, so a VM can boot from an attached ISO or other rescue media without reordering its disks. Unset keeps the previous positional boot order.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Configure a boot priority for each VM disk. Lower values are tried
first; disks without an explicit priority use their position in the disk
list.
* Disk priorities must be unique, and values below 1 are rejected.
Priority changes do not reorder disks or their associated volumes.
* **Documentation**
  * Added guidance and an example for configuring disk boot priorities.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:40:31 +03:00
Aleksei Sviridkin 8741fb31f9 fix(vm-instance): keep an external PortList VM valid with no ports (#5270)
## What this PR does

A `vm-instance` with `external: true`, `externalMethod: PortList` and an
empty `externalPorts` list renders a `LoadBalancer` Service with no port
at all. The API server rejects it (`Service "vm-instance-<name>" is
invalid: spec.ports: Required value`), so the HelmRelease fails on every
retry. The previous Service stays in place meanwhile, which means the
ports the user believes they removed are still open.

Emptying the list is a legitimate request: keep the VM's external IP
(egress SNAT, ICMP) but expose no inbound port. This PR renders a single
placeholder port in that case, `65535/SCTP`, named `no-ports`. The
protocol is the point: cozy-proxy's `EnsurePortFilter` only admits TCP
and UDP ports into `allowed_ports` and skips anything else, while the
pod still enters `filtered_pods`, so every inbound TCP and UDP packet is
dropped. A TCP or UDP placeholder would open that port on the VM
instead.

The `WholeIP` branch and the non-external branch keep their existing
`65535` placeholder unchanged.

Tested on a lab cluster running cozy-proxy with node-local rules,
against a running PortList VM exposing 22 and 80: with the placeholder
Service, MetalLB kept the same IP, cozy-proxy logged `Skipping
unsupported protocol for port filter` and kept the egress SNAT and
ingress DNAT, TCP 22, 80 and 65535 timed out from outside, ICMP still
answered (`allowICMP: "true"`), and the Cilium agents logged nothing
about the Service. Restoring the port list reopened 22 and 80.

`tests/service_ports_test.yaml` covers the three external cases. Its
first case fails on `main` (`unknown path spec.ports[0]`) and passes
with the fix.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (`templates/service.yaml` and a
new unit test in `packages/apps/vm-instance`): no package is added or
renamed, no value, schema, default or annotation changes, and the
`service-proxy-name` label and the `wholeIP` / `allowICMP` annotations
cozy-proxy matches on are untouched. The fix does rely on cozy-proxy
ignoring non-TCP/UDP ports in its port filter, which holds on cozy-proxy
`main` today; if it ever learns SCTP, this placeholder would open
`65535/SCTP` and nothing else.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-instance): an external PortList VM with an empty port list no longer fails to deploy; it keeps its external IP and blocks all inbound TCP and UDP traffic.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* External services without configured ports now include an SCTP
placeholder on port 65535, except for `WholeIP` services, whose existing
behavior is unchanged.
* Services with configured ports continue to render those ports without
the placeholder. This keeps the rendered service configuration
consistent across these cases.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:20:42 +03:00
Aleksei Sviridkin b6f5a4fa35 feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does

Adds a default scrape class `application-identity` to both VMAgents: the
cluster-wide one in `monitoring-agents` and the per-tenant one in
`monitoring`. Every series scraped from a pod that carries the lineage
labels `apps.cozystack.io/application.{group,kind,name}` gets them as
`apps_cozystack_io_application_{group,kind,name}`.

Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Design 4a)
needs to tell which application an exporter series belongs to. Today
that is only possible indirectly, through pod names.

How the labels are protected against forgery by an exporter:

- At the target stage, the class always sets all three labels: to the
pod's real lineage value, or to a placeholder `-` when the pod has none
(`-` is not a valid Kubernetes label value). With `honorLabels: false`,
any copy the exporter publishes therefore collides with the target label
and is renamed to `exported_*`.
- At the metric stage, the class drops every `exported_*` copy.
- The placeholder is removed by the VMAgent's global remote-write
relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent
adds `up` and `scrape_*` with the target labels after metric relabeling,
so only the remote-write stage sees those series. Targets without
lineage therefore keep exactly the label set they have today on every
series, including `up` and `scrape_*`.
- With `honorLabels: true` the exporter's value wins and no relabel rule
can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and
its series describe Ingress objects in any namespace, so the
controller's identity would misattribute traffic; its scrape now drops
the identity labels explicitly, from the target (so `up` and `scrape_*`
stay unchanged) and from the scraped samples (so an exporter's copy
cannot survive). Switching it to `honorLabels: false` was not done,
because that renames `namespace` and breaks the existing ingress
dashboards.

#### How it was tested

- helm-unittest for `monitoring-agents`, `monitoring` and `ingress`.
These tests check the rendered config only; the relabeling behaviour was
verified offline by running vmagent v1.136.0, built from source, against
the rendered rules: `up` and `scrape_*` carry no identity for targets
without lineage, and the placeholder appears on no series.
- The rendered `scrapeClasses` validate against the vendored VMAgent CRD
schema.
- The rendered rules were exercised on a locally built vmagent v1.136.0
(the version the bundled operator deploys) writing to a local
VictoriaMetrics: with and without lineage, with and without a forging
exporter, `honorLabels` true and false, and the ingress-nginx endpoint
rules. Checked on exporter samples and on `up`, `scrape_samples_scraped`
and `scrape_series_added`; the previous rules reproduce the `-`
placeholder on the generated series of targets without lineage, the new
ones leave no placeholder on any series.
- Verified on a test cluster with the earlier revision that removed the
placeholder at the metric stage (the move to remote-write relabeling is
not yet verified on a cluster): series carry the application labels;
metrics read for a Redis instance through the per-application proxy root
contained only that instance's pods; vmagent logs show no errors.

#### Known limitations

- Scrapes with `honorLabels: true` other than ingress-nginx are not
protected against an exporter that publishes the identity labels itself.
Today these are the vendored qdrant and harbor ServiceMonitors; their
exporters are platform images with fixed label names, and their series
describe the application's own pods.
- A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes
entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth
settings from a scrape class (operator v0.68.4); no VMProbe exists in
the platform today.
- A scrape object's own relabel rules run after the class and can
overwrite the labels; tenants cannot create VictoriaMetrics operator
objects, so this is platform-controlled.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff changes templates and tests in `packages/system/monitoring-agents`,
`packages/system/monitoring` and `packages/extra/ingress` without
adding, renaming or removing a package and without touching any
`values.yaml` or schema. The telemetry metrics the telemetry server
reads (`cozy_cluster_info`, `cozy_application_count`) are produced by
the operator, not by these scrapes. The proposal text in
`cozystack/community` (#94, Design 4a) should be corrected to name
qdrant and harbor as further `honorLabels: true` scrapes; that is not a
trigger-map item.

### Release note

```release-note
feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* Monitoring metrics now use application identity labels derived from
Kubernetes pod metadata for covered scrape targets. Exporter-supplied
duplicates and placeholder values are removed, including from `up` and
`scrape_*` metrics.
* Ingress metrics no longer retain application identity labels at either
target or metric relabeling stages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:03:57 +03:00
Andrey Kolkov 906eeb2372 feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does

Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.

None of the existing strategies fits a tenant cluster:

- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.

**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):

1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.

**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:

- **Application gone**:
  1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.

Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.

Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:

- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.

Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.

A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.

Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.

### Verification

- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
  - the full backup path;
  - in-place restores with default options and etcd only;
  - a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
  - RestoreJob deletion resuming a stopped cluster;
  - the stop deadline;
  - the preflight: missing or incomplete Velero backup, missing dump;
  - a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.

Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
  - a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
  - keys are moved from the old prefix to the new one;
  - an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.

### Not in this PR

- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.

### Release note

```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 15:40:01 +07:00
Mattia Eleuteri 9e861c1178 fix(seaweedfs): stop the master crashing on the crowded volume map
SeaweedFS 4.45 updates the crowded-volume map of a volume layout under
the read lock, while Assign reads that map under the same read lock to
count writable volumes. Under a steady write load the Go runtime aborts
the master with "concurrent map read and map write", a fatal error that
recover cannot catch, and the S3 gateway answers 500 until a master is
elected again. Taking the write lock for the update, as every other
mutation of the map already does, removes the race.

Upstream fixed it in seaweedfs/seaweedfs#11216, released in 4.46. The
patch carries that change and its regression test unchanged, so the
build runs the test and the patch stops applying, and is dropped, once
the vendored release carries the fix.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
2026-11-13 09:19:25 +01:00
myasnikovdaniil 0f169540a1 feat(release): withhold partial backports and park those that build on code their line lacks (#5269)
## What this PR does

Stacked on #5264, that one merges first.

Clean cherry-pick doesn't mean the original landed. On release-1.6 the
batch carried a fix whose two deletions hit files the line never had, so
only its test arrived. Other originals conflicted on every run because
they change code that never reaches the line. Batch now tells these
cases apart instead of picking half a fix or retrying a conflict nobody
can resolve.

- Every file an original modifies or deletes is checked at its
counterpart on the line, renames followed on both sides. If part of the
original has no effect there (change to a file the line lacks, deletion
of a file it never had, deleted file the unit keeps), the original is
parked as partial and its release note stays out. A file that already
matches what the original makes counts as there (same blob, binary too,
or clean `git merge-file`). This runs in the picker, `fold`, `verify`
and over units already on the open batch, so such a unit fails `check
--cut` whatever its label until `run --defer` takes it out.
- Pick that differs from its original keeps its release note under
"Check these notes" with a `git range-diff` command to compare, record
and report mark it `changed`.
- Run names what an original builds on that the line lacks: the commit
on `main` that added a file the fix modifies or deletes, and blame of
removed lines whose text is still missing on the line. Generated files
and generated keys of ApplicationDefinitions are skipped. Lines a fix
only adds are not checked, a dependency that lives only there still
needs `--drop`.
- If a prerequisite never reaches the line (`feat`, direct push,
excluded or itself parked) the original is parked and not tried. If it
is still to come, the original waits for it and is retried later.
Prerequisites the line or open batch already carry, or this run picked,
count as there. `run`, `check` and record-only commands decide in the
same order. On scheduled run a workflow original whose only missing
dependency is in tests or docs waits for a maintainer instead of
parking. `backport/include` lifts neither park, wait nor partial.
- `notify` announces each park once on the open batch and marks it in
the record, nothing is announced while no batch is open.

Record moves to version 3 and report to format 2, format 1 reports still
read. I ran the new tool next to the previous one without `--push` on
today's history. On release-1.7 nothing changes category and `check`
prints the same. On release-1.6 #5071 is parked on the two features it
builds on, #5088 already on the open batch now fails the cut as a
partial unit, and #5142 on scheduled run still waits for a maintainer
with a note that only its tests depend on what 1.6 lacks. Rule 6 holds
twelve originals on 1.6 right now, run over them directly flags the
seven that change code 1.6 never got and none of the other five.
Detection adds under 2 s per line.

After this merges, CI on the open 1.6 batch goes red at its next push
since `verify` refuses #5088 there. `run --line release-1.6 --defer 5088
--reason ... --push` takes it out.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(release): backport batch holds back a pick that applies only part of its original and fails the cut on such unit already on a batch, sets apart the note of a changed pick, parks an original that builds on code its line never gets, waits for one whose prerequisite is still to come, and announces each park once.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backport batches now identify partial picks and missing prerequisites,
and can park affected originals or leave them waiting.
* Batch reports and notifications provide details about parked, partial,
changed, and waiting picks.
* Release notes for picks that differ from their originals are set apart
for review.
* **Bug Fixes**
* Verification, folding, and batch checks reject picks that omit effects
from the original; renamed paths are supported.
* **Documentation**
* Updated backport guidance to explain parking, prerequisites, partial
picks, and release-note checks.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 13:14:12 +05:00
Aleksei Sviridkin c1652a730d fix(linstor): update LINSTOR server to v1.35.3
piraeus-operator v2.12.1 ships LINSTOR 1.35.3 as its controller and
satellite version, and the in-repo piraeus-server build has to match it.

LINSTOR 1.34 raised the minimum Java to 21, so the build and runtime
stages move from bookworm to trixie, and the system Gradle used by
debian/rules follows the upstream wrapper to 8.14.2.

Six carried patches are part of upstream by v1.35.3 and are dropped;
the stale-bitmap adjust retry is rebased onto the new DrbdAdm.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:37 +03:00
Aleksei Sviridkin e1ed8c7719 fix(linstor): update linstor-csi to v1.13.2
piraeus-operator v2.12.1 is released against LINSTOR CSI v1.13.2, and
the release line carries IO QoS limits through the qos.linbit.com/*
StorageClass parameters.

The operator gives the CSI pods their client certificate as mounted
files through LS_USER_CERTIFICATE_FILE and LS_USER_KEY_FILE, instead
of inline values in LS_USER_CERTIFICATE and LS_USER_KEY. The
linstor-wait-until v0.3.1 in this image is built on a golinstor
release that reads only the inline variables, so its init containers
present no client certificate, the LINSTOR API closes the TLS
connection, and they retry on EOF forever. Move to
linstor-wait-until v1.0.1, the version upstream v1.13.2 pins.

The Protocol C override for dual attach is dropped. Since v1.12 the
driver no longer sets allow-two-primaries itself: it asks LINSTOR to
manage dual-primary through make-available, and LINSTOR 1.35 sets
protocol C on the connections that need it. Upstream closed the
equivalent change as superseded for that reason
(piraeusdatastore/linstor-csi#435).

The relocate-after-clone patch is carried forward. Volume IDs became
a struct for consistency groups; Clone and VolFromSnap handle
standalone resources, so relocation keys on the resource name. The
enumer output is regenerated rather than hand-patched.

Upstream now pins grpc v1.84.0, which builds on Go 1.27 and includes
the fix for GO-2026-6443, so the grpc override is no longer needed.

The golden-image clone guard in vm-disk was re-checked: parameter
parsing for allowRemoteVolumeAccess and layerList, and the default
derived from the layer list, are unchanged in this range.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:36 +03:00
Aleksei Sviridkin acfae30136 fix(linstor): update piraeus-operator to v2.12.1
The nfs-server DaemonSet runs piraeus-csi-nfs-server v1.10.3, which
boots systemd as PID 1 in a privileged container with the host /dev
mounted. Its getty units hang up the host's getty@tty1 about a second
after the pod starts; systemd restarts the host getty until it hits
the start limit and gives up, so login on the local or out-of-band
console stops working on nodes that run the NFS server. Seen on
bare-metal Ubuntu 24.04 nodes with Cozystack 1.6.5 and 1.7.0.

Upstream fixed the image by disabling the services the NFS server does
not need (piraeusdatastore/linstor-csi#417, reported as
piraeusdatastore/piraeus-operator#961); v1.10.6 includes the fix.

v2.12.1 is the current stable release. Its image set is LINSTOR
1.35.3 and LINSTOR CSI v1.13.2, which this repository builds itself;
both builds follow in the next commits. The CSI release line is what
carries IO QoS limits (qos.linbit.com/* StorageClass parameters,
added in v1.11.3), which operators have asked for.

Behaviour that changes with this range:

- The CSIDriver moves to fsGroupPolicy: File, so fsGroup is applied
  to ReadWriteMany filesystem volumes too.
- The operator manages Secrets only in its own namespace, and the CSI
  controller loses its cluster-wide read on Secrets; nothing in this
  repository references snapshotter Secrets.
- The csi-health-monitor sidecar is removed.
- Volume group snapshots are enabled only when the
  groupsnapshot.storage.k8s.io/v1 API is served; the snapshot
  controller here still ships v1alpha1, so they stay off.

The csi-provisioner arguments are unchanged in this range, so the
restated strictTopology list stays complete.

The chart is vendored the way `make update` does it, which also drops
the stale CRD copy from the vendored chart; piraeus-operator-crds
installs the CRDs. Declare the local chart dependency so Helm can
validate the umbrella chart without changing its rendered resources.

This commit is not deployable on its own. The operator now hands the
CSI pods their client certificate through LS_USER_CERTIFICATE_FILE and
LS_USER_KEY_FILE, which the linstor-wait-until in the current
linstor-csi image does not read, so the CSI init containers retry
forever until the linstor-csi update in the next commit.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:36 +03:00
Aleksei Sviridkin 6297ebd708 fix(linstor): run LINSTOR init containers on the platform server image
piraeus-operator fills in its own default LINSTOR server image for every
container this package leaves unset, and two of those are init
containers: run-migration in the controller pod and
setup-lvm-configuration in the satellite pods. Only the main containers
were pinned to the platform build.

run-migration upgrades the LINSTOR database in place. When the operator
default is newer than the platform build, it migrates the schema ahead
of the controller, the controller then fails to read the migrated
database and crash-loops, and the migration cannot be rolled back. This
stayed hidden while the operator default was older than the version
built here; an operator update that moves the default past it exposes
it during the window between the operator and linstor releases
reconciling.

Pin both init containers to the same image as the main containers, so
the database moves only together with the controller.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 11:05:35 +03:00
myasnikovdaniil f061739ce2 ci(e2e): raise the E2E job limit from 215 to 265 minutes (#5288)
## What this PR does

Full chainsaw run doesn't fit in 215 minutes anymore. Last four nightly
runs were cancelled at the limit, and PRs that select full suite end
same way while every suite that got to run passed (#5283 for example).

This raises limit for all four jobs that run the suite (pull-requests,
e2e-fork, e2e-tag, nightly) from 215 to 265, so they stay on one figure.
Comments that state the cap follow it. Making e2e itself shorter is
separate work.

Part of #5261, `Collect report` ceiling in fork workflow from there is
not covered here.

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(e2e): E2E job limit raised from 215 to 265 minutes.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Increased the time limit for end-to-end test runs from 215 to 265
minutes across applicable workflows. This allows longer test runs to
complete before the overall job is stopped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 09:16:13 +03:00
scooby87 e978ed7f2f fix(postgres): stop backup jobs from fighting over the WAL archive destination (#5221)
## What this PR does

Part of #5202. A second PR will restore the last good destination when a
job that changed it fails.

The ObjectStore a Postgres Cluster archives its WAL to is shared by
every backup of the application, yet every running BackupJob server-side
applied its own strategy's destination onto it every five seconds with
forced ownership. Two BackupJobs of one application with different
BackupClasses flipped the destination back and forth until the healthy
backup failed too, and a webhook or apiserver blip during that apply
failed the BackupJob at once.

- A BackupJob writes the destination and the Cluster plugin only before
its `cnpg.io/Backup` exists. Afterwards it only reads the live Cluster,
and a failed read is returned for a controller retry instead of failing
the running backup.
- Before writing, a BackupJob waits, with a `Ready=False` /
`WaitingForOtherBackup` condition naming the other job, while another
running BackupJob of the application has a `cnpg.io/Backup` in flight.
The wait is not capped, for the same reason as in #5211. The queue reads
the Cluster's own Backups from the apiserver; with leader election one
reconcile runs at a time, so only a leader handover can still let two
jobs through. An atomic per-application claim is #4559.
- Before writing, a BackupJob also checks the apiserver for a restore of
its application and holds while one runs, including a job released from
the queue: a write landing during the restore would put the restored
Cluster back on the source's WAL prefix.
- Write errors that cannot succeed on retry (invalid, bad request,
forbidden) still fail the job at once. Any other is retried with a
`ClusterPluginApplyRetrying` condition for the backup deadline, counted
from the first failure since the job last waited, and recorded in a new
`DestinationApplied` condition.
- The pass that creates the `cnpg.io/Backup` records on it the
destination, endpoint, credentials Secret and serverName it wrote
(`backups.cozystack.io/cnpg-written-destination`), and the artifact and
its size probe are built from that record, so a strategy edited, or a
Cluster re-rendered, while the backup runs does not redirect the
artifact. A Backup without the record falls back to the strategy as
before.
- The plugin rollout deadline and the instance check count from the
write that attached the plugin, not from the job's start. The apply that
attaches it also stamps the time on the Cluster
(`backups.cozystack.io/barman-plugin-attached-at`), so a lost BackupJob
status write after the attach does not fall back to an older time, and a
job that starts while another job's recent attach is still rolling waits
for that rollout too. A stamp a few minutes ahead of the controller's
clock is waited out; one further ahead is treated as corrupt.

Tests drive the reconciler through a running backup that must not
rewrite the destination, the queue and its release, a restore starting
while a job waits, each class of write error, a recovered write, a
failed Cluster read while the Backup runs, lost status writes around the
attach, another job's rollout, clock drift on the attach stamp, the
recorded destination and serverName, and the rollout deadline. Each
guard was checked by mutation.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): Postgres BackupJobs of the same application no longer overwrite each other's WAL archive destination: a job writes it only before its backup starts and waits while another backup or a restore of the application is in flight. A transient error while writing it is retried instead of failing the job at once, and the backup artifact names the destination and serverName the backup was actually written to.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Reliability**
* CNPG backups wait while another backup or restore is using the shared
destination, and retry temporary destination-write failures while
observing the backup deadline.
* Backup artifacts and size checks use the destination recorded when the
backup was created, even if destination settings later change.
* Backup processing handles plugin rollout timing and destination-hold
release more consistently.
* **Restore**
* Restore operations account for shared WAL archive destination
conflicts and guard against incompatible backup selections.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:55 +03:00
Aleksei Sviridkin 2cf80c02a4 fix(clickhouse): keep failed backup attempts out of the bucket (#5229)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse BackupJob attempt that fails during upload leaves its local
backup on the data volume, and with the upstream defaults the
clickhouse-backup sidecar never prunes local backups and resumes every
unfinished upload when it starts. On the next sidecar restart the failed
attempt is uploaded to the bucket as an archive no Backup refers to, and
while that resumed upload runs the sidecar answers new operations with
423, which burns the strategy Job's retries in seconds. The strategy
itself only waited for the sidecar API to respond and then started its
operation regardless.

The sidecar now runs with `API_COMPLETE_RESUMABLE_AFTER_RESTART=false`,
so a failed attempt is never published, and `BACKUPS_TO_KEEP_LOCAL=-1`,
so at most the newest local backup is kept and it is removed once its
upload succeeds. Backups already left on existing volumes are cleaned up
by the first backup after the upgrade. The Altinity strategy now waits
until the sidecar has no operation in progress before it starts a backup
or a restore, and fails the attempt with a clear message instead of
proceeding when the sidecar stays busy or unreachable.

Waiting for an idle sidecar changes what a retried strategy Pod does.
The Job retries a failed Pod up to twice, and a Pod can die after its
operation was accepted (a dropped poll under errexit, an eviction, a
node drain) while the sidecar keeps running it. Before, the retry hit
423 and failed; with the idle gate it would wait for the running
operation to finish and start it again. For a restore that is
destructive, because `restore_remote` drops every table it restores
before recreating it, so the second run discards whatever was written
after the first one finished. A retried restore therefore takes over a
`restore_remote` of the same archive that is still running and waits for
its result instead of starting another. Once an operation is accepted,
polls tolerate up to six failures in a row before the attempt fails, and
the sidecar losing the operation from its in-memory action log, or
ending it cancelled, fails the Pod instead of leaving it polling.

A retried backup works from the archive name the controller records for
its BackupJob. That name is now `<application>-<YYYYMMDDHHMMSS>-<first
12 hex digits of the BackupJob UID>`, so two runs of one application
started in the same second, as two Plans on one schedule are, never
share an archive. It is exactly as long as the previous
`clickhouse-<application>-<YYYYMMDDTHHMMSSZ>`, so an `s3PathOverride`
prefix that fit before still keeps every object key within the 1024-byte
limit. A retry takes over a `create_remote --resume` of its archive that
is still running, counts an upload of its archive whose latest run ended
in `success` as its own result instead of uploading the archive again,
and otherwise resumes the upload. Restoring a Backup that records no
archive still picks an archive by timestamp, and that path accepts only
`clickhouse-<application>-<timestamp>` names, so it never picks a
per-run archive.

One limit remains. A finished `restore_remote` row is not taken over:
the sidecar never trims its action log, so a `success` row of the same
archive may be left by an earlier RestoreJob, and counting it as this
one would report a restore that never ran. A retry that arrives after
the restore it replaces had already finished, but before that Pod saw
the result, therefore still runs the restore a second time.

Upgrading changes the ClickHouseInstallation pod template, so ClickHouse
pods with `backup.enabled` are rolled once. A backup Job started before
the upgrade by a controller that named archives the previous way is not
re-rendered and keeps writing that name, so the controller records the
previous name when the new one is not on the Job. The local copy of the
latest failed attempt stays on the volume until the next backup. Objects
partially written to the bucket by an upload that fails midway are not
removed by clickhouse-backup and are left for a follow-up.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): failed ClickHouse backup attempts are no longer uploaded to the bucket when the backup sidecar restarts, local backups no longer accumulate on the data volume, the backup strategy waits for the sidecar to be idle instead of failing with HTTP 423, a retried ClickHouse restore takes over the restore that is still running instead of starting it a second time, and a retried ClickHouse backup takes over or reuses its own upload instead of starting a new one. Every ClickHouse backup run now writes an archive of its own, even when two runs start in the same second, under a name no longer than before. ClickHouse pods with backups enabled are restarted once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup and restore operations wait for the backup service to become
idle and join a matching operation already in progress rather than
starting a duplicate.
* Restore operations use the archive the Backup records and fail when it
is not in remote storage; only a Backup that records no archive falls
back to the newest eligible archive by timestamp.
* New backups use distinct archive names, including for runs started in
the same second.
* Local backup copies are discarded after upload, and completed uploads
are not resumed after a service restart.
* **Bug Fixes**
* Operations now fail when the service cannot be queried, an operation
disappears from its activity log, or it ends with an error or
cancellation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:33 +03:00
Aleksei Sviridkin bc268ce759 fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first (#5152)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Disabling the Monitoring Agents addon of a tenant Kubernetes cluster
leaves `<cluster>-monitoring-agents` (release `cozy-monitoring-agents`,
installed into the tenant cluster) in `UninstallFailed` for good. Seen
in production on 1.6.5 and still reproducible from `main`.

`monitoring-agents`, `victoria-metrics-operator` and
`vertical-pod-autoscaler-crds` in
`packages/apps/kubernetes/templates/helmreleases/` are all gated on
`addons.monitoringAgents.enabled`, so they disappear together and Flux
uninstalls them in parallel: `dependsOn` is not honoured on deletion.
Two things then break the uninstall of `monitoring-agents`:

1. The VictoriaMetrics operator is often gone first. The operator's
`apps.victoriametrics.com/finalizer` stays on the `VMAgent`, nobody
removes it, and the deletion hangs. The manual workaround was to strip
the finalizer, after which GC removes the VMAgent's Deployment, Service,
Secrets and ServiceAccount through their owner references.
2. The VPA CRD was deleted with its own release while the stored
manifest still contains `VerticalPodAutoscaler vpa-vmagent`, so Helm
cannot build the manifest to uninstall. That root cause is fixed by
#3586 (`helm.sh/resource-policy: keep` on the tenant addon CRDs), which
this PR does not redo.

This PR:

- adds a `pre-delete` hook (`templates/vm-cleanup-hook.yaml`) that
deletes every VictoriaMetrics object the release owns, waits up to 60s
for a live operator to finalize them properly, and clears their
finalizers by hand when it has not. Objects are selected through Helm's
`meta.helm.sh/release-name` / `release-namespace` annotations rather
than by name, so the 34 `VMRule`s rendered from `alerts/` and the
`VMServiceScrape` in `cozy-victoria-metrics-operator` are covered along
with the VMAgent. The kinds handled are every VictoriaMetrics kind the
chart renders: `VMAgent`, `VMRule`, `VMServiceScrape`, `VMPodScrape`,
`VMNodeScrape`. Operator v0.68.4 only finalizes the VMAgent; the other
kinds are covered in case another operator version finalizes them, and
for them the patch is a no-op. A kind whose CRD is already gone is
skipped, and the script always exits 0, since a failing pre-delete hook
would itself block the uninstall. The Job is bounded by
`activeDeadlineSeconds: 180` and `backoffLimit: 0`; its kubectl calls
carry no `--request-timeout`, which would make kubectl skip the
in-cluster credentials (`hack/check-hook-request-timeout.bats`) and
reuses the digest-pinned `docker.io/clastix/kubectl:v1.32` image the
etcd-operator hook uses, via a new `kubectlImage` value. The shape
follows
`packages/system/forklift/templates/forklift-cr-cleanup-hook.yaml`.
- guards `templates/vpa.yaml` on `.Capabilities.APIVersions.Has
"autoscaling.k8s.io/v1/VerticalPodAutoscaler"`, so a cluster that does
not serve the CRD never records a VPA in the release manifest. Cheap
defence in depth next to #3586.

Points for review:

- The hook's RBAC is a ClusterRole/ClusterRoleBinding, limited to `get,
list, delete, patch` on the five `operator.victoriametrics.com`
resources, rather than Roles. The objects span two namespaces, and a
Role in `cozy-victoria-metrics-operator` could not be created if that
namespace were already gone or terminating, which would fail the hook
and the uninstall with it. All hook objects carry
`hook-succeeded,hook-failed,before-hook-creation` except the Job, which
keeps a failed pod for its logs (`ttlSecondsAfterFinished: 600`), as in
the forklift hook.
- When the operator is already gone, clearing the finalizer skips
whatever the operator does on finalize, notably the cluster-scoped RBAC
it creates for the VMAgent, which has no owner reference. That residue
is the same the manual workaround left, and it is no worse than today's
stuck release; removing it would mean hard-coding the operator's naming
scheme for those objects, which I preferred not to do.
- When the operator is alive, the hook only brings forward deletions
Helm was about to make, so the operator's normal finalization still
runs.

Tests: `tests/uninstall_test.yaml` asserts that the hook is a
`pre-delete` Job in the release namespace with a bounded runtime, that
its RBAC is hook-scoped and grants exactly the verbs needed on every
VictoriaMetrics kind the chart renders, that the binding targets the
hook's ServiceAccount, that the script selects by release ownership,
deletes, clears finalizers and exits 0, and that the VPA renders only
when the capability is present (using `capabilities.apiVersions`). With
the hook removed and `vpa.yaml` reverted, 6 of the 7 tests fail; with
the fix, `helm unittest packages/system/monitoring-agents` passes 26/26.

`hack/monitoring-agents-vm-cleanup-hook.bats` runs the rendered script
against a fake `kubectl` that keeps the objects in a state file, with
fake `date` and `sleep`. It checks that only objects annotated for this
release are deleted and patched, that with the operator gone the hook
polls through the grace period and then clears the finalizers, that a
live operator's objects are never patched, that a missing CRD still
exits 0 and cleans the other kinds, and that a release with nothing left
exits 0 without deleting anything. Removing the ownership condition,
making the patch a no-op, or inverting the polling condition each turns
this file red. The hook has not yet been run on a live tenant cluster.

Backport: labelled `kind/backport` (release-1.7). The release-1.6
backport is done by hand in #5158. The commit cherry-picks onto
`release-1.6` with a single trivial conflict in `values.yaml` (context
only: 1.6 lacks the `prometheus-node-exporter` resources block the new
`kubectlImage` block sits after on `main`). Resolved locally, the
chart's tests pass on 1.6 (9/9); 1.6's chart renders the same
VictoriaMetrics kinds. Note that `release-1.6`'s monitoring-agents
`Makefile` has no `test:` target, so CI does not run this chart's unit
tests on that branch.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff (a system chart's templates,
`values.yaml` and tests): no package added or renamed, no platform
values, no app schema, no CRD, no tooling or convention change. I also
searched cozystack/website for a documented VMAgent finalizer workaround
that this would make obsolete and found none.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring-agents): disabling the Monitoring Agents addon of a tenant Kubernetes cluster no longer leaves the monitoring-agents release stuck in UninstallFailed when the VictoriaMetrics operator is removed first; a pre-delete hook now releases the VictoriaMetrics objects' finalizers, and the VPA is only rendered when its CRD is served.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* VictoriaMetrics resources belonging to a release are cleaned up during
uninstall, including resources that remain because of finalizers.
Cleanup is limited to resources owned by that release.
* Vertical Pod Autoscaler resources are rendered only when the cluster
supports the required API.
* **Configuration**
* Added an image setting for the uninstall cleanup job, including a
pinned image version and pull policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 08:58:10 +03:00
Alexey Artamonov 583f18437d test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps
its archive name, and a run's archive name now carries part of the
BackupJob's UID, so the fixture needs one for the name it expects to
be the name the controller mints.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:21:51 +03:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Alexey Artamonov 5c6fc10a23 test(postgres): pin three guards no test caught when changed
A job whose own Backup exists skips the wait for another job's plugin
rollout, and the size probe keeps the strategy's Secret key names while
it takes the Secret from the record, adding a credentials block only
when the strategy has none. Each of these could be changed with the
suite still green: dropping the own-Backup check, or forcing the
missing-credentials branch either way. Cases for a job with a running
Backup next to a recent foreign attach, a strategy without credentials
and a strategy with custom key names now fail on each change.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:02:40 +03:00
Aleksei Sviridkin 238b3ee491 fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but
only the `s3_lifecycle` job of its plugin worker applies it, and that
job is scheduled by the admin server. Cozystack deployed neither, so a
lifecycle rule returned 200 and no object ever expired, on the platform
backup bucket and on tenant buckets alike. That also made the documented
way to bound backup storage a no-op.

This PR adds the admin server and a worker to Simple and MultiZone
SeaweedFS instances, behind a new `lifecycle.enabled` value that is
**off by default**. A Client instance is unchanged, because its rules
are enforced where the instance it connects to runs. The worker
registers for `s3_lifecycle` only, so vacuum, balancing and erasure
coding are not started as a side effect. The admin keeps its job state
on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default
when that is empty; `volume.storageClass` is not reused, because in
MultiZone it can name a class local to one zone or node. The admin UI is
protected by a generated password, because a tenant namespace admits
traffic from the whole cluster. The dashboard Role lists the two new
WorkloadMonitors.

Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart
renders no `[grpc.s3]` section, so the s3 gateway served its internal
gRPC in plaintext while the worker dials it with mTLS, and every
lifecycle delete failed the TLS handshake. The section is backported
from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart
patch, applied with `--forward` so the next `make update` stops at the
point where the patch must be dropped instead of silently reversing it.
The s3 pods are restarted once so they load the section. This also
repairs IAM cache propagation between s3 replicas, which failed in the
same way, and it applies whether or not lifecycle is enabled.

The data-plane release now retries a failed install or upgrade in place
(`RetryOnFailure`) instead of rolling it back. With the previous default
remediation and unlimited retries, an upgrade that waited out its
timeout, for example on a tenant quota with no room for the admin
volume, rolled back and retried forever, restarting the s3 pods twice
per cycle. The worker's hostname anti-affinity is relaxed to a
preference so a rollout finishes on a single-node cluster.

#### Why lifecycle is off by default

The daily walk that SeaweedFS uses to expire objects sends its deletes
without an object identity, and the s3 server skips its compare-and-swap
when no identity is given; it does not re-check the object's age either.
A key overwritten between the walk's listing and the delete loses its
new content. This path is unchanged from 4.45 through 4.52 and on
upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`,
`identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup
buckets write unique keys and are barely exposed; a tenant bucket that
rewrites a fixed key is. On top of that, turning enforcement on would
start applying, on upgrade, every rule that was stored and ignored until
now. Both are retention changes an operator should choose, so nothing
changes for existing instances until `lifecycle.enabled` is set. Making
it the default again is left to a separate PR that carries object
identity on the walk path; upstream report:
https://github.com/seaweedfs/seaweedfs/issues/12451.

#### Turning lifecycle on and off

When `lifecycle.enabled` is set, rules that have been stored but ignored
start applying within minutes, and objects they have already expired are
deleted on that first pass, so review existing bucket lifecycle rules
first. Set `lifecycle.storageClass` before enabling, and prefer a
replicated class: the class is part of the admin StatefulSet's volume
claim template, which cannot change, so changing it while lifecycle is
on makes every upgrade of the instance fail until it is reverted. Moving
the volume means disabling lifecycle, deleting the
`admin-data-seaweedfs-admin-0` PVC, changing the class and enabling
lifecycle again. Each instance gains two small pods and a 1Gi PVC; when
the tenant quota has no room for them, the data-plane release keeps
retrying in place until it has.

When `lifecycle.enabled` is turned off again, the parent release removes
the admin credentials Secret while the data-plane release removes the
admin StatefulSet only on its own reconcile, shortly after. An admin pod
that restarts in that window fails to start on the missing Secret until
the StatefulSet is gone; nothing is lost, and the order is left as is.
The admin PVC stays until the instance is deleted, and a later re-enable
starts the admin with a new password.

Verified on a test cluster with the 4.45 chart: without the backport,
the worker found the expired objects and every delete failed with `tls:
first record does not look like a TLS handshake`; with this branch and
lifecycle enabled, a rule with a past expiration date removed the
matching objects and left the others, both on a manual run and on the
first scheduled pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up needed, not opened yet:
`operations/services/backup-classes.md` (v1.7 and next) advises an S3
lifecycle policy, which the bundled SeaweedFS applies only once
`lifecycle.enabled` is set; the SeaweedFS reference page gains
`lifecycle.enabled` and `lifecycle.storageClass` from the regenerated
README on the next release
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The terraform provider does not model the SeaweedFS application, so the
new values do not reach it.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added optional, daily S3 bucket lifecycle-rule processing for
supported SeaweedFS configurations. It is disabled by default; Client
instances rely on lifecycle rules configured on the connected SeaweedFS.
* Added configurable storage for lifecycle job state and dashboard
visibility for lifecycle components.
* Enabled the S3 gRPC server to use its internal TLS identity when
security is configured.

* **Bug Fixes**
* Improved storage-class handling so numeric-looking names remain
strings and unset classes are omitted where appropriate.
* Clarified backup cleanup guidance, including when lifecycle rules
apply and how to reclaim storage.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:21:52 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Alexey Artamonov 1d22013646 fix(postgres): hold a backup until the plugin sees its new destination
The plugin-barman-cloud sidecar reads the ObjectStore through its own
cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud
v0.13.0) and invalidated only by the sidecar's own writes, and WAL
archiving keeps it warm. The pass that changed the destination created
the cnpg.io/Backup a moment later, so the base backup could be taken
against the previous destination while the artifact names the new one,
and a restore from it finds nothing. On a live cluster with the shipped
plugin, a Backup created two to four seconds after switching the
destination landed in the old bucket in two of three tries; with a
fifteen second wait it landed in the new one every time.

The apply that changes the ObjectStore configuration now stamps the
time on the ObjectStore itself, so the record cannot be lost apart from
the change, and the job creates its Backup only once that change is
older than the cache can be. A change is judged on the configuration a
server-side dry run of the same apply returns, so values the apiserver
defaults or another field manager owns do not read as a change on every
pass.

That wait leaves a job with the destination written and no Backup yet,
which the queue, looking only for Backups in flight, did not see: a
second job with another destination wrote its own, the first took that
as a change and stamped again, and the two restarted each other's wait
forever. Every apply of the ObjectStore therefore also names the job
that made it, which only a job without a Backup of its own and let
through by the queue does, and the queue holds every other job while
that one runs and has no Backup yet. Once it has one, that Backup holds
the destination for as long as CloudNativePG runs it, as before, so a
job stuck after its Backup finished does not hold every later backup of
the application. This also covers a job waiting for the
rollout of the plugin it attached, during which a second job used to
switch the destination as well, including when its own write left the
configuration as a restore had re-rendered it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 15:13:58 +03:00
Aleksei Sviridkin aa97451bc2 fix(platform): leave tenant-root's ingress empty when the root tenant has none (#5226)
## What this PR does

Enabling SeaweedFS on the root tenant without ingress no longer hangs.
`cozystack-basics` wrote `_namespace.ingress: tenant-root` for the root
tenant whatever its `spec.ingress` said, so `seaweedfs-system` depended
on an `ingress-nginx-system` release that nothing creates, and the
platform backup bucket waited with it. Every other tenant already
derives the key in `packages/apps/tenant/templates/namespace.yaml` and
leaves it empty without ingress.

`cozystack-basics` cannot read the root Tenant, so the platform now
carries its ingress state the same way it carries monitoring: the new
platform value `ingress.rootEnabled` reaches the `cozystack-values`
Secret as `_cluster.ingress-enabled`, and `_namespace.ingress` is
written only when it is true. The default is `true`, which keeps the
value every existing cluster renders today, and a Secret written before
the key existed keeps it as well. An operator running the root tenant
without ingress sets `ingress.rootEnabled: false`, kept in sync with
`Tenant/root spec.ingress` by hand, as with `monitoring.rootEnabled`.

With the key empty, `seaweedfs-system` drops the dependency through the
check it already had (`eq $ingress .Release.Namespace`), and nothing
reads live cluster state for it.

An empty `_namespace.ingress` now reaches tenant-root and every tenant
inheriting from it, so the consumers that would break on it are adjusted
in the same PR. The API rejects `ingressClassName: ""` (checked with a
server-side dry run), and two consumers quote the value:

- `kubernetes`: the proxied ingress-nginx addon Ingress is not rendered
without an ingress; its Service still is.
- `monitoring`: Grafana's `spec.ingress` is left out, since
grafana-operator copies it into the Ingress verbatim.
- `seaweedfs`: the S3 Ingress is disabled instead of rendering with no
controller to serve it.

The `KamajiControlPlane` `network.ingress` block is unchanged on
purpose: its `hostname` is what the Kamaji provider uses as the control
plane endpoint, and Kamaji itself skips an empty class name. Harbor,
bucket, bootbox and alerta write the class unquoted, so an empty value
renders as an unset class, which the API accepts.

Tests:
`packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml`
pins the flag reaching `_cluster` both ways and with `ingress` set to
null;
`packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml`
pins `_namespace.ingress` for a missing, string, bool and `"false"`
flag, and that the other keys stay when ingress is off;
`packages/extra/seaweedfs/tests/ingress_dependency_test.yaml` pins the
dependency and the S3 Ingress for an own, empty and inherited ingress;
the kubernetes and monitoring suites pin the gated Ingress. Reverting
any of the gates, or flipping the missing-key default, fails the suites.

Not covered here:

- `_namespace.etcd` has the same hardcoded shape (#4783) and is left for
its own change.
- Toggling `Tenant/root spec.ingress` without changing
`ingress.rootEnabled` leaves the value stale, as
`monitoring.rootEnabled` does today.

### Screenshots

Not a UI change.

### Downstream repositories

The diff adds a key to `packages/core/platform/values.yaml`, which the
website documents by hand.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#800 documents `ingress.rootEnabled` in the
platform Package reference
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a root tenant without ingress no longer leaves seaweedfs-system waiting for an ingress-nginx-system release that is never created. Set the new platform value ingress.rootEnabled to false to match Tenant/root spec.ingress: false.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a root-tenant ingress setting, enabled by default, that can be
turned off.
* **Bug Fixes**
* Kubernetes and Grafana ingress resources are now rendered only when
ingress is configured.
  * SeaweedFS S3 ingress is enabled only when ingress is configured.
* Ingress-dependent components now respect the root-tenant setting,
while other platform services remain available when root ingress is off.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 15:06:12 +03:00
myasnikovdaniil 76a34d3e53 fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix (#5284)
## What this PR does

This PR bumps seaweedfs COSI driver to v0.4.0, because v0.3.1 can lose a
grant for good: BucketAccess reports `accessGranted` and its Secret is
written, but S3 gateways keep rejecting the key with
`InvalidAccessKeyId`. Fixes #4912.

Driver v0.3.1 grants access by rewriting legacy
`/etc/iam/identity.json`, and every S3 gateway migrates this file to
per-identity files and renames it away on each change, so grant written
during migration is dropped. v0.4.0 also writes
`/etc/iam/identities/<user>.json` on grant
(seaweedfs/seaweedfs-cosi-driver#15). First version of this PR built the
driver from that upstream commit, now upstream released v0.4.0 so it's
just a tag bump.

Rest of v0.4.0 doesn't reach the platform:

- `readwrite-governance-bypass` access policy
(seaweedfs/seaweedfs-cosi-driver#16) is opt-in, none of our
BucketAccessClasses use it.
- Bucket deletion is refused while objects hold an active lock
(seaweedfs/seaweedfs-cosi-driver#18), only for object-lock buckets. All
`-lock` BucketClasses are `Retain`, and sidecar calls DriverDeleteBucket
only for `Delete`.
- Filer errors (seaweedfs/seaweedfs-cosi-driver#20) are now matched by
exact not-found message, I checked it against seaweedfs 4.45 filer and
formats match. One behavior change: bucket deletion doesn't ignore
errors of the recursive delete anymore, so sidecar retries instead of
reporting success.

How I tested (seaweedfs 4.45, two S3 gateways): Bucket with one user,
then three more users added in one change, after a minute ListObjects
with each user key, rejected key retried once more after 30s. Five runs
for each driver, the fixed one is a build of `5efd120`, the commit
v0.4.0 takes the grant fix from:

| driver | keys rejected for good |
| --- | --- |
| v0.3.1 | 5 of 20 (4 runs out of 5 lost at least one) |
| `5efd120` | 0 of 20 |

Same failure broke v1.7.1-rc.1 e2e, RabbitMQ backup round-trip got `403
InvalidAccessKeyId` for freshly granted key:
https://github.com/cozystack/cozystack/actions/runs/42980741995

Keys already lost on running clusters stay broken after upgrade.
Workaround from #4912 still works: remove user from Bucket, wait until
its BucketAccess is gone, add it back.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(seaweedfs): bucket users granted together could get S3 keys that never work (403 InvalidAccessKeyId). COSI driver is bumped to v0.4.0, which writes each grant to per-identity IAM file.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Updates**
  * Updated the SeaweedFS COSI driver to version `v0.4.0`.
* The updated driver writes IAM files per identity, helping preserve
grants when identity files are updated.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:31:44 +05:00
Myasnikov Daniil c63e83466a ci(e2e): raise the E2E job limit from 215 to 265 minutes
A green run of the full Chainsaw suite no longer fits in 215 minutes:
the last four nightly runs were cancelled at the limit, and pull
requests that select the full suite end the same way while every
suite that got to run passed. All lanes that run the suite keep one
figure, so a change that passes on a pull request is not cut off on
nightly or on a tag.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 15:42:25 +05:00
Aleksei Sviridkin 0b284a65d5 test(release): isolate report deduplication from workflow attempts (#5282)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Pins the report deduplication test to the first workflow attempt. In a
CI rerun, the inherited attempt otherwise adds `/attempts/2` to the
synthetic run URL and fails the existing exact-URL assertion. Production
run URLs and the deduplication assertions stay unchanged.

Validation: a source-derived check of the test setup and original URL
assertion fails with an inherited second attempt and passes with the
proposed first-attempt setup. Controls with the pin after preview or set
to 2 still fail. The full Git-backed test was not run locally; CI must
verify it.

### Screenshots

Not applicable; no UI changes.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(release): make the report deduplication test independent of the surrounding workflow attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* Updated automated test configuration to account for GitHub Actions run
attempts. No user-facing behavior changes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 13:01:17 +03:00
Alexey Artamonov 5a95bcdd64 fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024
bytes, and the legacy s3PathOverride prefix has no length limit. The
full UID made the name 37 characters longer than before, so a prefix
that fit the old name pushed the manifest key past the cap and failed
the backup.

The run's name now drops "clickhouse-" and the "T" and "Z" of the
timestamp and spends the room on the first 12 hex digits of the UID,
so it is exactly as long as the old name for every application.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 11:10:43 +03:00
Myasnikov Daniil 6a93765351 fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix
seaweedfs-cosi-driver v0.3.1 grants access by rewriting the legacy
/etc/iam/identity.json. Each S3 gateway migrates that file to
per-identity files and renames it away whenever it changes, so a grant
written while a gateway is migrating is lost: the BucketAccess reports
accessGranted and its Secret is written, but every gateway rejects the
key with InvalidAccessKeyId and never recovers. Users added to a Bucket
in one change lose a key regularly. v0.4.0 also writes the per-identity
file on grant.

The rest of v0.4.0 does not reach the platform. The
readwrite-governance-bypass access policy is opt-in and no
BucketAccessClass uses it. The refusal to delete a bucket that still
holds locked objects applies only to object-lock buckets, and their
classes are Retain, so the sidecar never calls DriverDeleteBucket for
them.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 07:53:34 +00:00
Alexey Artamonov 67fcadf689 fix(seaweedfs): disable the S3 Ingress in a tenant without ingress
The S3 Ingress kept rendering with no ingress-nginx to serve it. Now
that seaweedfs-system no longer waits on ingress-nginx-system there, the
release completes and the unserved Ingress would stay behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov 275c840a8c fix(monitoring): leave Grafana's ingress out in a tenant without ingress
grafana-operator creates the Ingress from spec.ingress verbatim, so an
empty _namespace.ingress reached the API as ingressClassName "" and was
rejected. With no ingress-nginx in the tenant there is nothing for that
Ingress to attach to.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov c61e6a842e fix(kubernetes): skip the proxied addon Ingress in a tenant without ingress
An empty _namespace.ingress means no ingress-nginx serves the tenant,
which tenant-root now reports when the root Tenant has ingress off. The
Ingress would carry ingressClassName "", which the API rejects, and the
release would fail to install where it used to leave an unserved
Ingress behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov b26001b6f9 test(seaweedfs): pin the ingress-nginx dependency to the namespace's ingress
seaweedfs-system waits on ingress-nginx-system only when _namespace.ingress
names its own namespace. With tenant-root's key now empty when the root
Tenant runs no ingress, an empty value has to drop the dependency, and a
value inherited from a parent must not add it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:36 +03:00
Alexey Artamonov 5b6708c7f9 fix(platform): leave tenant-root's ingress empty when the root has none
cozystack-basics wrote _namespace.ingress: tenant-root whatever the root
Tenant's spec.ingress said, so applications in tenant-root were told an
ingress-nginx runs there when nothing deploys it. SeaweedFS then waited
forever on ingress-nginx-system, and the platform backup bucket with it.
Every other tenant already derives the key and leaves it empty without
ingress.

cozystack-basics cannot read the root Tenant, so the platform carries
its ingress state the way it carries monitoring: ingress.rootEnabled
reaches the Secret as _cluster.ingress-enabled and gates the key. The
default keeps the value existing clusters render, and a Secret written
before the key existed keeps it too.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:35 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Aleksei Sviridkin 9fe06a874b test(release): isolate report deduplication from workflow attempts
CI reruns expose GITHUB_RUN_ATTEMPT to a test that expects a URL for
the first attempt. Keep its synthetic run identity independent of the
surrounding workflow.

Fixes: 5a997fb7d4 ("feat(release): post an immutable report of every classification and write record version 2")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-11 15:35:18 +03:00
Alexey Artamonov 35823c6c9c test(clickhouse): list the system-bucket round-trip in the suite header
The header names the backup Tests of the suite and why they are split;
it still described two of them after the third was added, so a reader
starting from it would miss the only Test that covers the system-bucket
path.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:26 +03:00
Alexey Artamonov 8ba0380043 docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches
S3, among them the create_remote, restore_remote and list calls the
strategy makes. The status and action-log endpoints the strategy polls
read in-memory state and do not reload it, so the mounted file reaches
the sidecar on the calls that need credentials rather than on every
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:11 +03:00
Andrey Kolkov 3b4db69f8e feat(backups): record the RDB snapshot location and size for Redis backups (#5046)
## What this PR does

Builds on the shared S3 probe from #5223 (MariaDB and CNPG), which is on
`main`. Refs #4316.

Backups made by the Redis strategy carry no `Backup.status.artifact` at
all: no URI and no size. This covers both the Redis and Valkey apps,
which share the driver. The snapshot is a single object per backup,
`<namespace>/<app>/<backup>.rdb` (`redisObjectKey`). The driver already
records that key in `driverMetadata`. The default strategy uploads the
object using the S3 coordinates from the backup credentials Secret,
which the controller projects into the application namespace
(`endpoint`, `bucketName`, `region` and the access keys).

When a backup completes, the driver now:
1. reads that projected Secret (`CredentialsConfig.TargetSecretName`) in
the application namespace. The projector stores its `endpoint` without a
scheme, and the strategy script reaches it over https; an endpoint with
a path prefix is refused by minio-go, which leaves the artifact unset;
2. reads the TLS trust from the pod spec of the completed backup Job, in
the order the script applies it: `ca.crt`, then `tls.crt`, from the
Secret mounted as the `s3-ca` volume
(`backupStorage.endpointCASecretName`). Only when there is no `s3-ca`
volume does it honour `S3_INSECURE_SKIP_VERIFY=true`, and only when
every container of the pod sets it, since the probe cannot tell which
container uploaded. The Secret is read at probe time; the default
strategy mounts `s3-ca` only when an endpoint CA is configured, so when
the volume yields no CA now (the Secret is gone or lacks both keys, or
the volume is not a Secret), the probe leaves the artifact unset rather
than trust the system roots or skip verification where the Job may have
verified. The CA is trusted on its own, as `curl --cacert` does, not on
top of the system roots;
3. runs `StatObject` on the recorded key; a response without
`Content-Length`, which minio-go reports as size -1, counts as
unmeasured;
4. records `status.artifact.uri = s3://<bucket>/<key>` and
`status.artifact.sizeBytes` = the object's size. An empty object is
recorded too, but `sizeBytes` is `omitempty`, so on the Backup it reads
like an unmeasured size; since a valid RDB snapshot always carries a
header, a Warning `EmptySnapshot` Event on the BackupJob flags it as a
broken backup.

The artifact is set **only when the object is found**. A custom strategy
that writes elsewhere would otherwise get a URI pointing at an object
that does not exist. On any failure (Secret missing or incomplete,
endpoint invalid, CA unreadable, object not found, TLS error) the
artifact stays absent and the backup still succeeds. The full error is
logged at info, and once the Backup is created a Normal
`ArtifactNotMeasured` Event on the BackupJob gives a fixed reason, also
when a failed existence check skipped the probe; the Event lives in the
application namespace, so it never carries the raw error, which may name
the platform's endpoint and bucket. The probe shares the 10s timeout of
the MariaDB and CNPG probes. The retry check is shared with those
drivers too: a `Backup` that already exists is returned without probing,
a failed existence check creates the Backup without probing, a `Create`
collision is resolved by reading past the cache, and an existing Backup
that cannot be read back is retried with `Ready=False` instead of
failing the BackupJob.

The size is useful because an RDB snapshot is the compressed dataset. It
shows growth, and a snapshot that is suddenly tiny is a broken backup.

Kafka is deliberately left out. Its backup is a topic-metadata export of
a few kilobytes, so the size would say nothing about the data and might
suggest the data itself was backed up.

Tests:
- the stat uses the projected credentials;
- every path that leaves the artifact unset: object missing, Secret
missing, bucket, endpoint, access key or secret key missing, no target
Secret configured, a size the endpoint did not report;
- trust resolution: `ca.crt` wins over `tls.crt` and skip-verify;
`items` remapping of the volume, and a key the `items` do not project is
never trusted; the Redis CA volume mounted next to `s3-ca` is not taken
as the S3 CA; an `s3-ca` volume that yields no CA (Secret gone, Secret
without the keys, a ConfigMap source) is refused with and without
skip-verify; skip-verify applies only on an exact `true` of that
variable in every container, the last entry of a repeated variable
winning;
- the real minio client against a TLS stub: its certificate is trusted
only through the `s3-ca` volume or skip-verify, a CA that did not sign
it is refused even with skip-verify set on the Job, and the shared
client enforces a configured CA with and without the system roots; the
client sends a signed `HEAD` and parses `Content-Length`;
- the Events: an unmeasured artifact or a failed existence check raises
`ArtifactNotMeasured` without the raw error; an empty snapshot is
recorded and raises a Warning `EmptySnapshot`; a measured artifact, a
rejected `Create` and a `Create` collision raise none;
- a reconcile test pinning that the completed Job's pod spec reaches the
probe; a Backup that already exists is returned without a probe or a
`Create`; the `Create` collision is read back from the API server past
the cache.

The suite fails if any guard of the trust lookup (volume name, Secret
source, variable name, every-container rule, refusal of an unreadable
CA), the CA enforcement in the transport, the negative-size check, the
Events, the reason-only message, their place after `Create`, the
empty-snapshot warning, the existing-Backup check, the wiring, any term
of the Secret check, or the API-server read-back is removed.

### Screenshots

N/A — no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is Go-only in
`internal/backupcontroller`, with no API, CRD, chart, value or schema
change. Existing `Backup` fields are now filled for one more driver.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(backups): Redis and Valkey backups now record `status.artifact` with the `s3://` location and the size of the RDB snapshot. If the snapshot cannot be found with the platform backup credentials, the artifact stays unset and a Normal `ArtifactNotMeasured` Event on the BackupJob says why. An empty snapshot raises a Warning `EmptySnapshot` Event.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Redis backups now include artifact location and size when they can be
measured.
* Backups are still created when artifact measurement fails, with status
reporting that the artifact was not measured.
* **Bug Fixes**
* Improved handling of existing backups and temporary read-back failures
to support reliable reconciliation.
* Improved compatibility with S3 endpoints using custom certificate
authorities or configured TLS verification settings.
  * Zero-size snapshots are now reported with a warning.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 15:52:54 +04:00
Alexey Artamonov c26d2773a5 test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever
BackupJob ran in tenant-root before this Test, from this suite or
another: Chainsaw does not fix the order of the Tests in a suite, so no
particular one can be named. The restore budget is also not spent on a
credentials hold: by then the stamp is older than the backup's own
two-minute wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:42:11 +03:00
Aleksei Sviridkin fb5ac1826a test(observability-proxy): prepare seed stubs in both unit runners
The POSIX runner does not invoke Bats setup hooks. Initialize the
capture stub from the test body so the compatibility lane stays offline.

Fixes: 5a2582e21c ("test(observability-proxy): preserve JSON seed request bodies")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-11 12:12:11 +03:00
Aleksei Sviridkin 66acdb80e1 ci(e2e): skip final cleanup on disposable PR runners (#5260)
## What this PR does

Remove the final sandbox teardown from the two PR E2E workflows. The
runners are documented as disposable VMs, so destroying their local
containers and zpools after testing delays the result without protecting
a later run. Cleanup between preparation attempts and the Makefile
targets for reused hosts stay unchanged.

The job limit stays at 215 minutes. Suite-length budgeting, including
nightly and tag runs, remains separate in #5261. This change relies on
the existing disposable-runner contract; it does not change runner
provisioning.

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **CI Workflows**
* Removed automatic sandbox and cluster teardown after end-to-end report
uploads.
* Added checks that cluster cleanup runs before environment preparation
is retried.

These changes affect automated test workflows and do not alter
user-facing product features.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 12:10:26 +03:00