Commit Graph
1899 Commits
Author SHA1 Message Date
Aleksei Sviridkin 7c46e1e384 fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does

The barman-cloud sidecar of Postgres backups is deliberately left
without a CPU limit, so that a throttled sidecar slows WAL archiving
down instead of failing it. In a tenant with `resourceQuotas` set that
intent does not hold: `packages/apps/tenant/templates/quota.yaml`
creates the `tenant-range-limits` LimitRange, whose default CPU limit is
`250m`, and the sidecar receives it. Full backups and WAL segments are
then gzip-compressed on a quarter of a core. On a production tenant the
first full backup of a 1.6 GB database took 30 minutes, uploading at
about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small
database on the same destination finished in 35 seconds.

This sets an explicit `limits.cpu: "1"` on the sidecar, in the two
places that build its configuration: `barmanSidecarConfiguration` in the
backup driver (platform-managed ObjectStores) and
`cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the
postgres and keycloak charts). This mirrors how the memory request and
limit there already override the LimitRange's 128Mi default. One core
removes the 250m ceiling without letting the sidecar starve the
instance. Inside a tenant it counts against the `limits.cpu` quota, one
core per instance pod; the comments now say so.

The sidecar is injected when the instance pod is created, so existing
instances pick the new limit up at their next restart.

Tests: the Go test that asserted the absence of a CPU limit now asserts
one core, and so does
`packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail
with the implementation reverted and pass with it. `go test
./internal/backupcontroller/`, `helm unittest packages/apps/postgres`
(220) and `helm unittest packages/system/keycloak` (121) pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (the backup driver, `cozy-lib`'s
barman template, two tests): no app, value, schema, default, CRD,
annotation or tooling change, so none of the listed repositories
restates anything touched here.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart.
```
2026-11-13 14:54:51 +03:00
Aleksei Sviridkin 56224a156c feat(virtual-machine): let each disk set its own boot order (#5266)
## What this PR does

A `vm-instance` boots from its disks in the order they appear in
`disks`: the template hardcodes `bootOrder: {{ add $i 1 }}`. Booting
from installer or rescue media, such as an ISO attached as a CD-ROM next
to the system disk, therefore means moving that media to the front of
the list, and anything that treats `disks[0]` as the system disk then
finds the ISO there instead.

This adds an optional `bootOrder` to each entry of `disks`:

- When set, it is the disk's boot priority (1 is tried first),
independent of its position in the list.
- When unset, the disk keeps its positional order (`index + 1`), so
existing VMs render exactly as before.
- The values schema requires `bootOrder >= 1`.
- Duplicate effective boot orders are refused at render time with the
names of both disks, rather than surfacing as a KubeVirt admission error
on the VirtualMachine. This includes an explicit value colliding with
another disk's positional one, so booting from a CD-ROM in second
position means setting both disks:

```yaml
disks:
  - name: system
    bootOrder: 2
  - name: install-iso
    bus: sata
    bootOrder: 1
```

`make generate` was run in the package, so the README,
`values.schema.json`, the Go type in `api/apps/v1alpha1/vminstance`
(with its regenerated deepcopy, since the field is a pointer) and the
`vm-instance-rd` OpenAPI schema are all updated.

A new suite, `tests/disk_boot_order_test.yaml`, mocks the DataVolumes
the template looks up and covers: positional order when no disk sets
`bootOrder`; a CD-ROM in second position booting first while the list
order and volumes stay unchanged; an explicit value colliding with a
positional one; two equal explicit values; and a value below 1 rejected
by the schema. Run against the previous template, the CD-ROM and
collision cases fail. `helm unittest packages/apps/vm-instance`: 6
suites, 77 tests passing.

### Screenshots

No UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

This adds a field to `vm-instance`'s `values.schema.json`, which the
trigger map routes to `cozystack/terraform-provider-cozystack`: its
`vminstance_schema.go` and `vminstance_model.go` model `disks` by hand,
so `bootOrder` needs adding there. The follow-up has not been opened yet
and the box is left blank until it is linked. The website reference page
is regenerated from the README by the release bot; nothing else in the
map is touched.

### Release note

```release-note
feat(virtual-machine): disks in a vm-instance accept an optional `bootOrder`, so a VM can boot from an attached ISO or other rescue media without reordering its disks. Unset keeps the previous positional boot order.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Configure a boot priority for each VM disk. Lower values are tried
first; disks without an explicit priority use their position in the disk
list.
* Disk priorities must be unique, and values below 1 are rejected.
Priority changes do not reorder disks or their associated volumes.
* **Documentation**
  * Added guidance and an example for configuring disk boot priorities.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:40:31 +03:00
Aleksei Sviridkin 8741fb31f9 fix(vm-instance): keep an external PortList VM valid with no ports (#5270)
## What this PR does

A `vm-instance` with `external: true`, `externalMethod: PortList` and an
empty `externalPorts` list renders a `LoadBalancer` Service with no port
at all. The API server rejects it (`Service "vm-instance-<name>" is
invalid: spec.ports: Required value`), so the HelmRelease fails on every
retry. The previous Service stays in place meanwhile, which means the
ports the user believes they removed are still open.

Emptying the list is a legitimate request: keep the VM's external IP
(egress SNAT, ICMP) but expose no inbound port. This PR renders a single
placeholder port in that case, `65535/SCTP`, named `no-ports`. The
protocol is the point: cozy-proxy's `EnsurePortFilter` only admits TCP
and UDP ports into `allowed_ports` and skips anything else, while the
pod still enters `filtered_pods`, so every inbound TCP and UDP packet is
dropped. A TCP or UDP placeholder would open that port on the VM
instead.

The `WholeIP` branch and the non-external branch keep their existing
`65535` placeholder unchanged.

Tested on a lab cluster running cozy-proxy with node-local rules,
against a running PortList VM exposing 22 and 80: with the placeholder
Service, MetalLB kept the same IP, cozy-proxy logged `Skipping
unsupported protocol for port filter` and kept the egress SNAT and
ingress DNAT, TCP 22, 80 and 65535 timed out from outside, ICMP still
answered (`allowICMP: "true"`), and the Cilium agents logged nothing
about the Service. Restoring the port list reopened 22 and 80.

`tests/service_ports_test.yaml` covers the three external cases. Its
first case fails on `main` (`unknown path spec.ports[0]`) and passes
with the fix.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff (`templates/service.yaml` and a
new unit test in `packages/apps/vm-instance`): no package is added or
renamed, no value, schema, default or annotation changes, and the
`service-proxy-name` label and the `wholeIP` / `allowICMP` annotations
cozy-proxy matches on are untouched. The fix does rely on cozy-proxy
ignoring non-TCP/UDP ports in its port filter, which holds on cozy-proxy
`main` today; if it ever learns SCTP, this placeholder would open
`65535/SCTP` and nothing else.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-instance): an external PortList VM with an empty port list no longer fails to deploy; it keeps its external IP and blocks all inbound TCP and UDP traffic.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* External services without configured ports now include an SCTP
placeholder on port 65535, except for `WholeIP` services, whose existing
behavior is unchanged.
* Services with configured ports continue to render those ports without
the placeholder. This keeps the rendered service configuration
consistent across these cases.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 12:20:42 +03:00
Alexey Artamonov dae76d186c Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for
late bucket credentials, against the controller's archive helper.
That helper now names a run's archive with part of the BackupJob's
UID, so the merged test has to give its BackupJob one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-12 17:20:19 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Alexey Artamonov c61e6a842e fix(kubernetes): skip the proxied addon Ingress in a tenant without ingress
An empty _namespace.ingress means no ingress-nginx serves the tenant,
which tenant-root now reports when the root Tenant has ingress off. The
Ingress would carry ingressClassName "", which the API rejects, and the
release would fail to install where it used to leave an unserved
Ingress behind.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:20:38 +03:00
Alexey Artamonov 8ba0380043 docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches
S3, among them the create_remote, restore_remote and list calls the
strategy makes. The status and action-log endpoints the strategy polls
read in-memory state and do not reload it, so the mounted file reaches
the sidecar on the calls that need credentials rather than on every
request.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:11 +03:00
Mattia Eleuteri f1dd2d17fc fix(backups): give the barman-cloud sidecar an explicit CPU limit
The sidecar was left without a CPU limit so that a throttled sidecar
would slow WAL archiving down rather than fail it. In a tenant with
resourceQuotas that does not hold: the tenant LimitRange gives every
container a 250m CPU limit by default, so the sidecar gets it and
compresses full backups and WAL segments on a quarter of a core. A
1.6 GB database took 30 minutes to back up.

Set an explicit one-core limit in both places that build the sidecar
configuration, the backup driver and cozy-lib, the same way the memory
request and limit already override the LimitRange defaults.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-11 09:20:19 +01:00
Mattia Eleuteri c6f1d6c559 fix(vm-instance): keep an external PortList VM valid with no ports
The API server rejects a Service that declares no port, so a VM switched
to external PortList with an empty port list could not be upgraded: the
release failed and was retried forever, while the old Service kept the
previously listed ports open.

Render a single SCTP placeholder in that case. cozy-proxy only admits
TCP and UDP ports into its PortList filter, so the VM keeps its external
IP for egress and ICMP while all inbound TCP and UDP traffic is dropped,
which is what an empty port list means.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-10 14:37:16 +01:00
Mattia Eleuteri d48897cf82 feat(virtual-machine): let each disk set its own boot order
The boot order of a VM's disks was the order of the disks list, so the
only way to boot from installer or rescue media, such as an ISO
attached as a CD-ROM, was to move that media to the front of the list.
Callers that treat the first disk as the system disk then see the ISO
in its place.

An optional bootOrder per disk decides boot priority independently of
the list. Disks that leave it unset keep their positional order, so
existing VMs render unchanged. Duplicate effective boot orders are
refused at render time with the names of both disks, since KubeVirt
would otherwise deny the VirtualMachine with a less specific error.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-10 10:28:28 +01:00
Aleksei Sviridkin ba0312b35f chore(kubernetes): align monitoring fix with current platform base
The previous base differs from the published package configuration,
so the image overlay retains release controllers instead of current
builds.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 11:45:36 +03:00
Aleksei Sviridkin b53ca1fede fix(postgres): refuse system databases and never drop them (#5155)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

`databases` accepts `postgres`, `template0` and `template1`, and the
init job treats them like application databases: it comments them
`database managed by helm`, creates `<db>_admin` and `<db>_readonly`,
makes `<db>_admin` the owner of the database and its schemas, and
installs the `trigger_auto_grant` event trigger. When the entry later
leaves values, the delete pass runs `DROP DATABASE` on it, which can
only fail:

- `template1`, `template0`: `cannot drop a template database`;
- `postgres`: `cannot drop the currently open database`, since the job
is connected to it.

The init job then fails on every upgrade, so the HelmRelease sits in
`UpgradeFailed` with `post-upgrade hooks failed`, and putting the entry
back does not help. `template1` also hands the takeover (owner of
`public`, default privileges, trigger) to every database created
afterwards. Once its roles are dropped by hand, the trigger left behind
names a role that no longer exists, so `CREATE SCHEMA` fails in every
new database, even as superuser.

This PR:

- **Refuses** the three names at render with a message that says what to
do. `cozyvalues-gen` has no way to constrain the keys of a
`map[string]Database`, so the refusal sits next to the existing
database-name guard, like the `users.postgres` refusal. The field
description says it too.
- **Never selects them for `DROP DATABASE`.** The delete pass excludes
them whatever values say.
- **Hands a taken-over system database back.** In a new `release system
databases` pass before the delete pass, for each of the three still
commented `database managed by helm`: drop the trigger and
`auto_grant_schema_privileges()`, `REASSIGN OWNED` / `DROP OWNED` for
its two roles in that database, give `public` back to
`pg_database_owner` (PostgreSQL 15+), and clear the comment. The
existing orphaned-roles pass then drops the roles; it never visits
`template1`, hence the per-database step. `template0` accepts no
connection and the job that commented it never got further, so only its
comment is cleared.

**Upgrade note.** A release that declares one of the three names fails
to render until the entry is removed. Removing it is now safe: the next
run releases the database instead of dropping it.

**Testing**

- `tests/system_databases_test.yaml`: 5 of its 6 cases fail on `main`
without the change. `helm unittest packages/apps/postgres`: 215 passed.
- I replayed the scenario against a `postgres:17` container, running the
rendered `init.sh` with `PGUSER=postgres`:
1. The `main` script with `databases: {appdb, template1, postgres}`
reproduces the takeover.
2. The `main` script with only `appdb` then fails with `cannot drop the
currently open database`.
3. This branch's script on that same state exits 0. It releases both
databases (owner `postgres`, no comment, no event trigger, `public`
owned by `pg_database_owner`) and drops `postgres_admin`,
`postgres_readonly`, `template1_admin` and `template1_readonly`. A
database created afterwards gets no trigger, and `CREATE SCHEMA`
succeeds in it.
  4. A second run is a no-op.
5. A `template0` commented and given roles by hand is released the same
way.

Open PR #5091 touches the same script (stable database order); the hunks
do not overlap.

Backport: the automatic cherry-pick onto `release-1.6` conflicts in
`init-script.yaml` and the generated cozyrd, so the backport to
`release-1.6` is done by hand in #5157 (same SQL, replayed against that
branch's script). `release-1.7` keeps `kind/backport`.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change (walked the
map: a template guard, the init script, a test suite and one field
description regenerated into README, schema, Go types and cozyrd; no
field, default, enum or name change)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): databases now refuses the PostgreSQL system databases postgres, template0 and template1, and the init job never runs DROP DATABASE on them. A system database a previous version took over is handed back to the superuser on the next upgrade (event trigger removed, ownership and public schema restored, its <db>_admin and <db>_readonly roles dropped). A release that declares one of them must remove the entry before upgrading.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Behavior Changes**
* PostgreSQL system databases (`postgres`, `template0`, and `template1`)
are refused as managed application databases.
* Previously managed system databases are released during cleanup and
excluded from database deletion. Cleanup restores eligible ownership and
security settings and clears system database comments.
* **Documentation**
* Updated PostgreSQL configuration guidance to clarify that these system
database names are refused. Names that merely contain these strings
remain accepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 21:09:23 +03:00
myasnikovdaniil 91f03dd90e fix(platform): remove the dead registries value and patch-containerd Secret (#5190)
Nothing reads the `patch-containerd` Secret: the kubeadm certs.d mount
it fed is gone, and the v1.6.0 changelog already says the passthrough is
removed. The `registries` value stayed in the Package spec and takes
mirrors that do nothing, which is worse than no knob for someone setting
up an air-gapped cluster. Removed both.

A Package that still sets `registries` keeps installing, the platform
chart has no values schema so the key is ignored. On upgrade the Secret
in `cozy-system` is deleted by the release.

Mirrors for tenant workers are the per-pool `talos.registryMirrors` of
KubernetesNodes.

I checked that nothing is left: `git grep` for `.Values.registries` and
for `patch-containerd` outside the changelogs finds nothing but the
kubernetes README sentence that says the Secret was removed. Full `make
unit-tests` passes.

Fixes https://github.com/cozystack/cozystack/issues/4006


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration Changes**
* The platform chart no longer supports the `registries` configuration
option or generates containerd registry secrets. Mirror endpoint
settings, including credentials and TLS verification options, are no
longer applied through this chart.
* Registry mirrors can be configured per worker pool using
`talos.registryMirrors` on `KubernetesNodes`. Existing packages that set
`registries` continue to install, but the setting is ignored.
* **Documentation**
  * Updated guidance for configuring registry mirrors on worker pools.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 19:36:30 +05:00
Alexey Artamonov 89f8cc02ee Merge branch 'main' into fix/clickhouse-idle-before-backup
main now names the ClickHouse archive in the controller and restores
the archive a Backup recorded, in the same strategy script this branch
gates on an idle sidecar. The merged script keeps both: the archive
name and its restore selection from main, the idle gate, retried polls
and lost-operation check from this branch.

A retried backup is taken over through the same idle gate as a restore,
matched on the whole command line the sidecar logs. With resume
requested that line reads "create_remote --resume <name>", so the gate
matches that form; matching on the name alone would also take another
operation's row ending in the same name for this one.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 16:39:43 +03:00
Mattia Eleuteri f003851338 fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled
like an application database. The init job made <db>_admin their owner,
installed the auto-grant event trigger in them and, once the entry left
values, ran DROP DATABASE on them. That DROP can only fail (template
databases cannot be dropped, and the job is connected to postgres), so
every later upgrade failed in the init job and the release stayed in
UpgradeFailed even after the entry was put back. template1 also passed
the takeover on to every database created from it afterwards, where the
trigger, naming a role about to be dropped, makes CREATE SCHEMA fail.

A release a previous version let take one of them over has to be
repaired without dropping it, and without touching what this chart did
not create: a role or owner that merely carries an expected name may
belong to a tenant or an administrator, and a role comment can be set
by any CREATEROLE holder, so only the cozystack:managed mark is trusted.
What the marked roles own goes to cozystack:orphaned rather than to the
superuser, since a tenant's SECURITY DEFINER function would otherwise
run with superuser rights. A failed lookup of the databases cloned from
template1 has to stop the run before template1 stops being selected, or
those clones would never be repaired.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-09 11:18:27 +01:00
Aleksei Sviridkin 23461e065d fix(redis): run one Sentinel when Redis runs a single pod (#5151)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The chart pins `spec.sentinel.replicas: 3` whatever `replicas` says, so
an instance created with `replicas: 1` runs one Redis pod and three
Sentinel pods that have nothing to fail over to. This makes the Sentinel
count follow the data pods: one Sentinel when `replicas` is 1, three
otherwise.

The operator cannot run zero Sentinels (freshworks `Validate()` resets
`sentinel.replicas <= 0` to 3), and it needs at least one to locate the
master. With one Sentinel its quorum is `replicas/2+1` = 1, so the
single Sentinel works on its own.

The operator also gives the Sentinel Deployment a PodDisruptionBudget
with `minAvailable: 1` whenever Redis has two replicas or fewer
(`ensurePodDisruptionBudget` in
`operator/redisfailover/service/client.go`). On a single Sentinel that
budget refuses every eviction and would block node drains, so the chart
sets `sentinel.disablePodDisruptionBudget: true` in that case only. The
Sentinel `WorkloadMonitor` now expects the rendered count instead of a
fixed 3/2.

The operator only stops ensuring that PDB once the flag is set; it never
deletes the one it created while three Sentinels ran
(`EnsureSentinelDeployment` in freshworks v3.3.5). An existing instance
scaled to one replica would keep it, so
`templates/hooks/sentinel-pdb.yaml` adds a post-upgrade hook that
deletes `rfs-<release>`, with a Role scoped to that one name. It renders
on upgrade only, since a fresh one-replica install never gets that PDB.

From `replicas: 2` up nothing changes. Scaling an existing instance
between 1 and 2 replicas scales the Sentinel Deployment between 1 and 3
accordingly.

Not in scope: with `replicas: 1` the Redis data pod keeps its own
operator PDB (`minAvailable: 1`), which already blocks drains of its
node today; that is unchanged here.

Tests: `tests/sentinel_replicas_test.yaml` covers 1, 2, 3 and the
default, the PDB switch, and the `WorkloadMonitor`. Three of its cases
fail on `main` without the template change.
`tests/sentinel_pdb_hook_test.yaml` covers the hook. `helm unittest
packages/apps/redis`: 79 passed. Open PR #4990 also edits
`redisfailover.yaml` (Sentinel spreading); the two touch different keys.

Backport: `release-1.7` ships the same freshworks operator and CRD, so
`kind/backport` applies. `release-1.6` is not backportable as is: it
still runs spotahome `v1.3.0-rc1`, whose vendored CRD (chart 3.3.0) does
not declare `sentinel.disablePodDisruptionBudget`, and the operator's
CRDs are installed with Flux's default `crds: Create` policy, so they
are never updated on an existing cluster. The API server would prune the
field and the single Sentinel would keep a `minAvailable: 1` PDB that
blocks node drains. A 1.6 fix would need a CRD update path first.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change (walked the
map: the diff changes a template, a test, and the wording of two
descriptions; no field, default, enum or name changes)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(redis): an instance with `replicas: 1` now runs one Sentinel instead of three, without a Sentinel PodDisruptionBudget; on upgrade, the PDB the operator created earlier is deleted. Instances with two replicas or more are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Redis deployments configured with one replica now use one Sentinel;
other replica settings continue to use three.
* Workload monitoring now reflects the configured Sentinel count. When
changing to one replica, upgrades remove the Sentinel
PodDisruptionBudget.
* **Documentation**
* Updated Redis configuration guidance to explain how Sentinel count
varies with the replica setting and clarify Sentinel resource defaults.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:17:32 +03:00
Aleksei Sviridkin 0c486263f9 fix(postgres): render the declared databases in a stable order (#5091)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The postgres init script lists the declared databases with `keys
.Values.databases`, which returns them in Go's randomized map order. A
release with two or more databases therefore renders a different
`DEFINED_DBS` line on each upgrade, so the `<release>-init-script`
Secret and the `checksum/config` annotation of the init Job change on
upgrades that did not touch the databases. Sorting the keys makes the
render a function of the values alone.

Found while measuring a platform-wide values change on a production
cluster (#5092): of 36 releases that ran Jobs again, 31 rendered a
byte-identical manifest across the two Helm revisions and the other 5
were Postgres releases whose only difference was this ordering, e.g.
`DEFINED_DBS="db_a db_b"` → `DEFINED_DBS="db_b db_a"`.

This does not stop the init Job from running on every upgrade (it is a
`post-upgrade` hook); the issue covers that part.

Verification: the new `tests/init_script_deterministic_test.yaml`
renders eight databases declared out of order and expects them sorted.
Without the fix it failed 5 runs out of 5 with a different order each
time; with it, `helm unittest packages/apps/postgres` passes (27 suites,
210 tests).


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): the init script renders the declared databases in a stable order, so an upgrade that leaves the databases unchanged no longer rewrites the init-script Secret.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Database initialization now lists configured databases in alphabetical
order, providing consistent output across deployments.
* **Tests**
  * Added coverage to verify the alphabetical ordering.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:06:40 +03:00
Mattia Eleuteri 9fba8af69c test(redis): pin the release name in the Sentinel PDB hook for the longest release
The long-name case checked only that the Job name fits in 63 characters and that the Job targets the right PDB. A Job name that dropped the release, letting two releases' hooks replace each other, or a Role granting a different PDB would both have passed.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-08 19:45:18 +01:00
Mattia Eleuteri a7f84242d1 fix(redis): keep three Sentinels for a zero or negative replica count
The operator runs three Redis pods when the count is zero or negative, but the chart treated any count up to 1 as the single-pod case. Such a cluster got one Sentinel and no Sentinel PDB, while three Redis pods ran under it.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-08 19:36:44 +01:00
Mattia Eleuteri 55d6c52fab fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert
The monitoring-agents chart reads vmagent.remoteWrite.urls as a list and
takes the tenant label from global.target, while the kubernetes app still
passed a single remoteWrite.url and vmagent.externalLabels.tenant. Both
keys were dropped in silence, so the vmagent of every tenant cluster fell
back to the chart defaults: it wrote to vminsert services in
cozy-monitoring that do not exist inside the tenant cluster, queued the
failed writes on the node disk, and labelled every series
tenant="cozy-monitoring". No tenant cluster metric reached the tenant's
Monitoring.

The tenant label has to name the tenant that owns the cluster, which is
not always the tenant whose Monitoring receives the data: a tenant
without its own Monitoring inherits a parent's.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-08 19:33:54 +01:00
Alexey Artamonov 86c541368d Merge branch 'main' into fix/clickhouse-backup-late-credentials
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Myasnikov Daniil 41507e36aa docs(kubernetes): say where worker registry mirrors are configured
The air-gapped note said in-guest image pulls honour no per-tenant
mirror until a Phase 2 that has landed, so an operator setting up an
air-gapped cluster would read that mirroring is unavailable and file
an issue. It now says the Secret and the platform registries value are
removed, that a Package still setting the value keeps installing, and
that worker mirrors are set per pool through talos.registryMirrors of
the KubernetesNodes resource.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-06 18:56:09 +05:00
myasnikovdaniil e5bb3cda07 fix(kubernetes): move the OIDC users out of the bootstrap Job's script (#5213)
## What this PR does

Found while checking for other places with the failure from
https://github.com/cozystack/cozystack/issues/4968: data that grows with
configuration is passed to a process as one argument, and Linux allows
128 KiB for that. This one is in the `kubernetes` app and there is no
issue about it yet.

The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users`
entry into its `sh -c` script, 526 bytes per user. I rendered current
main with `helm template` and users like `user123@example.org`: 238
users give a 130,889 bytes script that starts, 239 give 131,415 bytes
and `exec` fails with E2BIG. Then the container can't start, the
post-install and post-upgrade hook fails after its retries and the
HelmRelease of the tenant cluster stays unready. Schema has no limit for
`oidc.users` and each person needs own entry, so a cluster shared by 240
people gets there.

Now the bindings and the list of their names are rendered by the new
`oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings`
that the Job mounts, so the script is 6,380 bytes for any number of
users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417
bytes each. Names, labels and roles of the bindings stay the same. I
compared the render of old and new chart for eight emails (ones YAML
reads as a boolean or null, a number, a quote, an apostrophe, unicode
and two plain) and the bindings are identical, so existing ones are
updated in place and nothing is orphaned.

The prune step reads the same ConfigMap. If the list is not mounted the
Job stops before it deletes anything, because `grep` on a missing file
exits 2, `! grep` is true and every binding of the release would be
pruned. A release with no users applies nothing (`kubectl apply` fails
on an empty file) and prunes all stale bindings, same as before.

`hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the
script is under the limit without any email in it and that every user
has one binding with the old name and role. Three more tests run the
rendered script against a fake `kubectl` and check what it applies and
deletes. The prune condition forced to true or to false passes all
helm-unittest cases, so this part is tested in bats. The bug is in
release-1.6 and release-1.7 as well.

### Downstream repositories

I walked the trigger map against the diff (one chart template, its tests
and a bats file): no package added or removed, no values, schema or
default changed, no output Secret or Service renamed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Kubernetes OIDC configuration now supports per-user role bindings,
with administrator and viewer access assigned according to each user’s
role.
* OIDC role bindings are kept in sync with the configured user list,
including removal of obsolete bindings.
* Deployments with no OIDC users or with OIDC mode disabled continue to
avoid creating user bindings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-05 19:37:16 +05:00
Alexey Artamonov 31b905a1ce docs(clickhouse): note that the backup path stays in the sidecar environment
The chart comment said every S3 setting of the system-bucket flow comes
from the mounted config file, but S3_PATH is still set in the
environment: the platform's file carries no path, and the prefix belongs
to the release rather than to the platform.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:51:38 +03:00
Alexey Artamonov 4f3f1a5cad fix(clickhouse): stop the backup sidecar from publishing failed attempts
create_remote keeps its local backup when the upload fails, and with
the upstream defaults the sidecar never prunes local backups and
resumes every unfinished upload when it starts. A strategy attempt that
fails mid-upload therefore lands in the bucket on the next sidecar
restart as an archive no Backup refers to, and restore can pick it as
the newest one. The resumed upload also holds the API lock, so the
strategy Pods racing it get 423 and burn the Job's retries.

Turning the resume off keeps a failed attempt out of the bucket, and a
negative local retention leaves at most the newest local backup and
drops it once its upload succeeds, so leftovers no longer accumulate on
the data volume.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:26:23 +03:00
Alexey Artamonov c9ce4c480f fix(clickhouse): let the backup sidecar pick up credentials projected after it started
With backup.useSystemBucket the clickhouse-backup sidecar took its S3
settings from optional secretKeyRefs on cozy-backups-creds. The platform
projects that Secret into the namespace at the first BackupJob, usually
after the ClickHouse pod started, and a container's environment is fixed
at start, so the sidecar ran without credentials and every backup and
restore failed until the pod was restarted by hand. Rotated keys stayed
stale the same way.

The projector now also writes the s3 section of a clickhouse-backup
config into the Secret, and the chart mounts that key, optionally, as
the sidecar's config file instead of setting S3 environment variables.
clickhouse-backup reloads its config file in each API handler that
reaches S3 and kubelet rewrites a mounted Secret in place, so projected
and rotated credentials reach the running sidecar. The flat keys stay
for the Redis strategy and for ClickHouse releases still rendered by
the previous chart.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 17:12:31 +03:00
Myasnikov Daniil 7b7444feaf fix(kubernetes): move the OIDC users out of the bootstrap Job's script
The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into
its script, and the script reaches the container as one `sh -c`
argument. Linux caps a single argument at 128 KiB, which about 240
users cross at roughly 530 bytes each. The container then cannot start,
the post-install and post-upgrade hook fails, and the tenant cluster's
HelmRelease stays unready. The values schema puts no bound on
oidc.users, so a shared cluster reaches that size with ordinary
per-person entries.

The bindings and the list of their names now reach the Job through a
ConfigMap it mounts, so the script is the same size for any list. Names,
labels and roles of the bindings are unchanged, so the bindings already
in a tenant cluster are updated in place and none is orphaned.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 07:06:31 +00:00
ZverGuy 95dc591198 test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a
trustworthy application identity, and the platform must not promote
them to the stream fields the parent scopes application logs by. The
guest release replaces the monitoring-agents filters and outputs
wholesale, which is what keeps the parent's label lift out of guests;
pin that so a later change to the guest overrides cannot bring it in.

This only covers what the platform ships. A guest admin can still edit
the guest's fluent-bit or write to the parent's vlinsert directly; that
needs a write-path control, which the logs proposal leaves open.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-04 04:18:09 +07:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Mattia Eleuteri 2541aec625 fix(redis): keep the Sentinel PDB hook Job name within 63 characters
The API server copies a Job's name into its pods' job-name label, which
takes 63 characters at most, and a release name may take 53, so
<release>-sentinel-pdb could be refused and leave the stale PDB in
place. The Job name is truncated; the ServiceAccount, Role and
RoleBinding keep the full name.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-02 11:32:58 +01:00
Mattia Eleuteri 7c1bd33b08 fix(redis): delete the Sentinel PDB left behind on scale-down
Setting sentinel.disablePodDisruptionBudget only stops the operator from
ensuring the PDB; it never deletes the one it created while three
Sentinels ran. An existing instance scaled to one Redis pod therefore
kept minAvailable 1 on its single Sentinel, which refuses every
eviction. A post-upgrade hook deletes rfs-<release> in that case.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-02 11:23:46 +01:00
IvanHunters 8d96c46994 fix(monitoring-agents): isolate merged log payload under log_processed
Fluent Bit's kubernetes filter runs with Merge_Log On but without
Merge_Log_Key, so a parsed JSON log line is merged into the record root.
An application log line can thereby forge the service fields that later
pipeline stages rely on for routing and tenant isolation
(kubernetes_namespace_name, kubernetes.labels, tenant, cluster,
log_source). Setting Merge_Log_Key log_processed nests the parsed
payload under a dedicated key so application content can no longer
override top-level service fields.

Applies to both the platform (system/monitoring-agents) and guest
(apps/kubernetes) Fluent Bit configurations.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-29 17:08:49 +03:00
Mattia Eleuteri 48779cc03b fix(redis): run one Sentinel when Redis runs a single pod
The chart pinned three Sentinels whatever the replica count, so a
single-pod instance paid for three pods that have nothing to fail over
to. One Sentinel is enough for the operator to locate the master: its
quorum is replicas/2+1, which is 1.

The operator gives the Sentinel Deployment a PodDisruptionBudget with
minAvailable 1 whenever Redis has two replicas or fewer. On a single
Sentinel that budget refuses every eviction and would block node
drains, so it is disabled in that case. From two Redis pods up nothing
changes: three Sentinels, budget kept.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-29 10:00:55 +01:00
Aleksei Sviridkin 3abdff5900 fix(kubernetes-nodes): refuse a partial resources spec on every pool (#4973)
The `resources.cpu`/`resources.memory` pair check in `nodegroup.yaml`
ran only when an `instanceType` was set, while the field description
(values.yaml, README, schema, the KubernetesNodes OpenAPI schema)
promised a render-time error for a partial spec on any pool. Without an
instancetype a cpu-only pool renders a VM with no memory, which KubeVirt
refuses when the Machine is created, and a memory-only pool silently
sizes the guest at KubeVirt's default of one vCPU.

- The check now runs on every kubevirt pool. The proxmox branch already
required both, so this is the kubevirt half of one rule, and the field
descriptions are true as written (no doc change). On a pool with no
`instanceType` the message ends in "or neither and set instanceType", so
it does not send the user straight into the next refusal.
- A kubevirt pool with neither an `instanceType` nor both resources
fails at render with a message naming both, the way `vm-instance` does.
No such pool could have passed admission (no memory), so this refusal
breaks nothing that runs today. This is the decision the issue asked
for; if the gate should stay narrower, this commit can be dropped on its
own.
- **Upgrade impact.** One shape that renders on main stops rendering:
`instanceType: ""` with only `resources.memory`. On main it gets
`domain.memory.guest` and no `domain.cpu`, so KubeVirt runs the worker
with one vCPU; with this PR it fails with the partial-spec message, and
the pool's HelmRelease sits in a failed upgrade until the values are
edited. To keep the current size such a pool needs `resources.cpu: "1"`.
Adding that line renames the machine template (for the test values,
`kubernetes-myk8s-md0-28d812` becomes `kubernetes-myk8s-md0-1ce103`, the
same name on main and on this branch), so the pool's workers are
replaced once. cpu-only with `instanceType: ""` is not in this class: it
renders a VM with no memory, which KubeVirt refuses, so it never ran.
- `cores:` printed an unquoted (float64) core count of one million or
more in exponent form (`1e+06`). It is printed as an integer now, with
the same bytes for every smaller whole number.

Tests: `tests/resources_partial_test.yaml` covers both halves with and
without an instancetype, the no-sizing case, both fields set, and the
float64 formatting. The existing unquoted-zero case in
`resources_cpu_test.yaml` now reads the field as unset on an
instancetype-sized pool (with a `kubernetesProvider` stub for the
lookup), and zero beside memory is pinned as a partial spec.

Byte stability, checked by snapshotting the `KubevirtMachineTemplate` on
main for nine spellings of `resources.cpu` and replaying the snapshots
on this branch: `"2"`, `2`, `2.0`, `"2.0"`, `"+2"`, `"1e1"`, `"010"`,
`999999` render identical bytes and template names. Only unquoted
`4294967295` changes, from `cores: 4.294967295e+09` to `cores:
4294967295`.

Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38
suites, 233 tests, 12 snapshots), and on a trial merge with current main
(39 suites, 237 tests).

Closes #4546. The leading-zero inconsistency from the issue's comment is
addressed separately, together with #4554.

### Release note

```release-note
fix(kubernetes-nodes): a partial `resources` spec (only `cpu` or only `memory`) is now refused at render time on every kubevirt worker pool, not only on a pool with an `instanceType`, and a pool with neither an `instanceType` nor both resources is refused as well. Action required: a pool with `instanceType: ""` and only `resources.memory` rendered before (KubeVirt ran it with one vCPU) and now fails to upgrade; add `resources.cpu: "1"` to keep its current size. Adding that line renames the pool's machine template, so its workers are replaced once. Whole-number core counts of one million or more are now printed as integers instead of in exponent form.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* KubeVirt worker pools now require either an instance type or both CPU
and memory values. Configurations with only one resource or no sizing
information are rejected.
* Whole-number CPU values render without exponent notation, including
values written with a decimal fraction.
* An unquoted zero CPU value is treated as unset when an instance type
is provided; without an instance type, pairing zero CPU with memory
fails validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 11:22:48 +03:00
Aleksei Sviridkin 60d1f53e82 fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets (#4974)
On the kubevirt substrate `resources.cpu` is written into
`domain.cpu.cores` as spelled and decoded as YAML, so `"010"` gives the
VM eight cores, while the cluster-autoscaler capacity annotation parsed
the same string as decimal ten
(`capacity.cluster-autoscaler.kubernetes.io/cpu: "9000m"` on a node that
offers 7000m). This is the inconsistency reported in #4554 and in the
comment on #4546.

- The YAML-decoded value the range check already computes is kept, and
on kubevirt the capacity annotation and the reserved-CPU check start
from it.
- The kubelet reservations are not touched. The worker machineconfig
(`_talosconfigtemplate.tpl`) sizes them from the raw string read as a
decimal, and the TalosConfigTemplate name is a hash of that config, so
reading the decoded value there would roll every leading-zero pool once.
Instead the annotation subtracts the reservations the machineconfig
really carries from the cores the VM really has: for `"010"` that is
8000m − 2 × 500m = `"7000m"`, the node's real allocatable, while `"8"`
keeps `"7200m"` (2 × 400m). Tests pin both sides: the annotation and the
bootstrap `configRef` name in `resources_cpu_test.yaml`, and
`systemReserved.cpu` / `kubeReserved.cpu` in the machineconfig in
`talos_worker_config_test.yaml`.
- No pool rolls. Checked by rendering `nodegroup.yaml` on main and on
this branch for nine spellings (`"2"`, `2`, `2.0`, `"2.0"`, `"+2"`,
`"1e1"`, `"010"`, `999999`, `4294967295`): the only difference is the
capacity annotation of the `"010"` case, so the
`KubevirtMachineTemplate` name and the TalosConfigTemplate `configRef`
stay as they are. The machineconfig template itself is not changed.
- Proxmox parses the quantity as decimal for the machine template and
for the reservations alike, so it never had the inconsistency and is
unchanged (pinned by a test).
- The `resources` description says the value is decoded as YAML, what
that means for a leading zero, and that the reservations still read it
as a decimal. README, schema, the RD and the Go type are regenerated
from it.

This closes #4554 without the migration and the render-time refusal it
proposes. A refusal at render breaks a stored pool on its next upgrade,
a pattern in the OpenAPI schema breaks every update of an existing CR,
and a platform-wide migration over every `KubernetesNodes` and parent
`nodeGroups` entry is a lot of blast radius for a spelling nobody has
reported in a live cluster. With this change the autoscaler plans for
the node the VM really is. What is left of a leading zero is the octal
reading itself and reservations sized from the decimal value, and both
are documented. The unquoted zero of the issue is already harmless: it
is falsy, counts as unset, and renders the same bytes as an absent
field. If a migration is still wanted, please reopen.

Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38
suites, 233 tests, 12 snapshots; pointing the annotation back at the
decimal string turns the `"010"` case red with `9000m`), `make generate`
(no diff), and a trial merge with #4973 (clean, 243 tests green).

Closes #4554


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* KubeVirt VM sizing and autoscaler capacity now reflect YAML-decoded
CPU values. For example, `"010"` represents eight VM cores, while
kubelet reservations interpret it as decimal ten. `"8"` and `"10"` yield
`7200m` and `9000m` of autoscaler capacity, respectively. Proxmox
continues to interpret `"010"` as ten cores, yielding `9000m` of
capacity.
* **Documentation**
* Clarified explicit worker CPU and memory sizing behavior, including
how leading-zero CPU values affect VM sizing, reservations, and capacity
calculations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 11:22:40 +03:00
Aleksei Sviridkin 44848b9c9d fix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digest (#4917)
## What this PR does

`v1.7.0-rc.1` and `release-1.7` ship the Proxmox
cloud-controller-manager as
`ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.15.0`, a bare
tag. That tag was re-pushed upstream on 2026-09-27: it used to name the
2026-08-06 build (`sha256:fbc553921145…`) and now names
`sha256:d81702998ae1…`, a binary whose gitVersion is `0.15.0` without
the leading `v`. component-base v0.37 parses that string in an `init()`
and panics:

```
panic: version string "0.15.0" doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"
```

The CCM Deployment crash-loops, no new Node loses
`node.cloudprovider.kubernetes.io/uninitialized`, and workloads on new
workers stay Pending. On a live Proxmox cluster this went unnoticed for
five days: the existing Nodes were already initialized, and the failure
only showed when the next new worker came up. Every fresh install pulls
the broken build, so as it stands v1.7.0 would be the first stable
release with the Proxmox substrate, shipping a CCM that cannot start.

Two commits, both cherry-pick onto `release-1.7` unchanged:

1. **`fix(kubernetes)`**: run v0.16.1, pinned as tag plus digest in
`images/proxmox-cloud-controller-manager.tag`. That is the same shape
the chart already uses for the KubeVirt cloud provider. v0.16 builds
with goreleaser and stamps the version with its `v`. The config the init
container writes still parses: the cluster entry moved to
go-proxmox-pool's `ClusterConfig` with the same `url`, `insecure`,
`token_id`, `token_secret` and `region` keys, and `features.provider:
capmox` still writes `proxmox://<SMBIOS UUID>`, the providerID capmox
puts on the Machine.
2. **`fix(kubernetes, proxmox-csi)`**: pin the CSI driver from the same
registry at its current v0.20.0, with no version change. The
management-side controller goes through
`images/proxmox-csi-controller.tag`. The tenant-side node plugin goes
through `repository` + `tag` in
`packages/system/proxmox-csi/values.yaml`, which also puts it in
`hack/lib/image-refs.sh` and so in front of the release multi-arch gate.
Pinning takes the version out of the vendored chart's hands, so
`hack/proxmox-csi-driver-pin_test.bats` now fails unless the chart's
`appVersion` and both pins name the same release.

All three digests are OCI indexes for `linux/amd64` and `linux/arm64`.
The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes through image promotion, which does
not move a tag.

### Testing

- `helm unittest`: `packages/apps/kubernetes` 31 suites / 346 tests,
`packages/apps/kubernetes-nodes` 37 / 223, all passing.
- bats: the new test, plus `image-pin-consistency`,
`promote-rewrite-tags_test`, `overlay-main-images_test`,
`nightly-mirror_test`, `promote-retag_test`, `stitch-multiarch_test`,
`bats-strict-setup`, `bats-runner-coverage`: 107/107. The new test was
checked to fail on a node tag at another version.
- `git cherry-pick -x` of both commits onto `release-1.7` (315b335cf)
applies cleanly, and the Proxmox unit suites and the new bats test pass
there.
- v0.16.1 compatibility was checked against its source, as described
above. It has also run on a live Proxmox cluster, including the
initialisation of a new worker; the results are in the comment below.

### Screenshots

Not a UI change.

### Downstream repositories

The diff touches no app `values.schema.json` or defaults, adds or
renames no package, and moves nothing under `hack/`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): the Proxmox cloud-controller-manager now runs v0.16.1, and the Proxmox CCM and CSI driver images are pinned by digest. The v0.15.0 tag referenced before was re-pushed upstream to a build that panics on start, which left every new Proxmox worker with the uninitialized taint.
```
2026-10-29 11:20:22 +03:00
Marian Koreniuk f7913329b2 fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets
On the kubevirt substrate resources.cpu is written into domain.cpu.cores
as spelled and decoded as YAML, so "010" gives the VM eight cores, while
the cluster-autoscaler capacity annotation parsed the same string as
decimal ten. The autoscaler then planned 9000m on a node that offers
7000m.

Keep the YAML-decoded value the range check already computes and start
the capacity annotation and the reserved-CPU check from it on kubevirt.
The kubelet reservations stay as they are: the worker machineconfig
sizes them from the decimal string, and the TalosConfigTemplate name is
a hash of that config, so changing them would roll every leading-zero
pool once. The annotation therefore subtracts the reservations the
machineconfig really carries (2 x 500m for "010") from the cores the VM
really has, which is the node's real allocatable. Neither template name
changes for any spelling, so no pool rolls. Proxmox parses the quantity
as decimal on both sides and is unchanged. The field description says
the value is decoded as YAML and that the reservations still read it as
a decimal.

This closes #4554 without the migration and the render-time refusal it
proposes: a refusal breaks stored pools on their next upgrade, a schema
pattern breaks updates of existing CRs, and a platform-wide migration of
every KubernetesNodes and nodeGroups entry is a lot of blast radius for a
spelling nobody has reported in a live cluster.

Closes #4554

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 21:04:27 +01:00
Marian Koreniuk 780b068df0 fix(kubernetes, proxmox-csi): pin the Proxmox CSI driver images by digest
The CSI controller in the management cluster was referenced by the bare
tag ghcr.io/sergelogvinov/proxmox-csi-controller:v0.20.0, and the node
plugin in the tenant by the vendored chart's default, its appVersion as
a bare tag. That registry is the one whose proxmox-cloud-controller-
manager:v0.15.0 was re-pushed to a build that panics, so a tag there is
not a statement of what runs.

Both now carry the v0.20.0 digest: the controller through
images/proxmox-csi-controller.tag, like the cloud-controller-manager, and
the node plugin through repository and tag in the proxmox-csi values,
which also makes it visible to hack/lib/image-refs.sh and therefore to
the release multi-arch gate. Both digests are indexes for linux/amd64
and linux/arm64.

Pinning takes the version out of the vendored chart's hands: a chart
bump would bring a new appVersion while both pins kept naming the old
release, and the controller and the node plugin would drift apart from
the templates and from each other without anything noticing.
hack/proxmox-csi-driver-pin_test.bats fails unless the chart's
appVersion and both pins name the same release.

The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes them through image promotion, which
does not overwrite a tag.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-28 18:38:35 +01:00
Marian Koreniuk 05a347e6f6 fix(kubernetes): run proxmox-cloud-controller-manager v0.16.1, pinned by digest
The Proxmox CCM was referenced by its bare v0.15.0 tag, and that tag no
longer names the image it named when the template was written. It was
re-pushed on 2026-09-27 (index sha256:d81702998ae1, replacing the
2026-08-06 build sha256:fbc553921145) to a binary whose gitVersion is
"0.15.0" without the leading v. component-base v0.37 parses that string
in the init() of its legacy metrics registry and panics:

  panic: version string "0.15.0" doesn't match expected regular
  expression: "^v(\d+\.\d+\.\d+)"

so the Deployment crash-loops, no Node ever loses the
node.cloudprovider.kubernetes.io/uninitialized taint, and workloads on a
new worker stay Pending. Any node that pulls the tag fresh gets it, which
is every fresh install.

v0.16 builds with goreleaser and stamps gitVersion as v{{.Version}}. The
configuration this template writes is read unchanged: the cluster entry
moved to go-proxmox-pool's ClusterConfig with the same url, insecure,
token_id, token_secret and region keys, and features.provider: capmox
still writes proxmox://<SMBIOS UUID>, the providerID capmox puts on the
Machine.

The reference now lives in images/proxmox-cloud-controller-manager.tag
as tag plus digest, the shape the chart already uses for the KubeVirt
cloud provider, so a moving tag can no longer change what runs.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-28 18:38:35 +01:00
Marian Koreniuk 0e77e2b10f fix(kubernetes-nodes): name instanceType in the partial-spec refusal
On a pool with no instanceType, "set both resources.cpu and
resources.memory, or neither" sent the user straight into the next
refusal, since neither sizing source is refused as well. The message now
ends in "or neither and set instanceType" on such a pool; a pool sized by
an instancetype keeps the old wording.

The tests anchor both spellings, so the suffix cannot leak onto, or drop
off, either kind of pool.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 17:06:39 +01:00
Aleksei Sviridkin ae9e7c4c64 test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on
`strategy.name: RetryOnFailure` for both actions, and the e2e
HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names
those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets
the same strategy for the same reason and had no pin.

This adds
`packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a
mirror of `csi_install_retry_test.yaml` rendered on the proxmox
substrate, and updates the guard comment that pointed at the gap. No
template change.

Verified with `helm unittest .` in `packages/apps/kubernetes` (32
suites, 349 tests).

Closes #4731


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Expanded automated coverage for Proxmox CSI installation and upgrade
retry behavior, including the retry interval and remediation retry
limits. The checks verify that retries continue according to the
configured settings during installation and upgrades. This update does
not change user-facing functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 18:30:06 +03:00
myasnikovdaniil 1ac83aebd9 fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates (#5086)
## What this PR does

After platform upgrade the tenant `gateway` HelmRelease fails with
`timeout waiting for: [TenantGateway/<namespace>/cozystack status:
'InProgress']`, retries every 30s without end (`RetryOnFailure`) while
holding a helm-controller worker, and parent tenant release that waits
on it fails too.

helm-controller waits on every object of the release with kstatus on
install and on every upgrade, and kstatus reads TenantGateway
`Ready=False/ListenersNotReady` as InProgress. TenantGateway stays there
while any listener certificate is pending, which depends on ACME or DNS
and on routes owned by other releases, so waiting longer doesn't help.
It passed before only because the wait runs inside a Helm action and the
install happened before those listeners showed up.

I added `healthCheckExprs` for `TenantGateway` with `current: "true"`
and pinned `waitStrategy: poller`, because expressions are evaluated
only under poller. Wait stays on. Flux returns InProgress while
`status.observedGeneration` lags `metadata.generation` before it reads
any expression, so release still waits until controller observed the new
TenantGateway spec, but not for listeners or certificates. tenant-root
renders same template and gets the change too. Gateway release that is
already stuck picks this up on its next attempt, after the current one
times out.

New helm-unittest suite
(`packages/apps/tenant/tests/gateway_wait_test.yaml`) pins the strategy
and the whole expression list, checks there is no `disableWait` on
install or upgrade, and that no gateway release is rendered when
`gateway` is unset.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tenant): the tenant gateway release no longer times out on upgrade while TenantGateway listener certificates are pending, it waits only until the controller has observed the current TenantGateway spec.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Gateway releases now determine readiness from the gateway resource’s
current status, rather than waiting for its listeners and certificates
to become ready. Releases still wait for the latest resource update to
be observed.
* **Tests**
* Added checks for gateway release readiness behavior and for tenants
configured without a gateway.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-27 11:47:15 +05:00
Aleksei Sviridkin 749d401899 fix(kubernetes, opensearch, foundationdb)!: restrict managed application images (#5085)
## What this PR does

Image overrides let tenants run arbitrary containers in management
cluster, including helpers with access to Talos CA secrets and
Kubernetes API credentials. This PR removes those overrides and uses
images shipped by the platform.

Replaces [the original
PR](https://github.com/cozystack/cozystack/pull/4287) for 1.7. Andrei
Kvapil's commits keep his authorship.

- Kubernetes and KubernetesNodes helpers use the bundled image
references through `cozy-lib.image`. OpenSearch uses its version map.
Talos installer settings for tenant worker VMs stay available.
- FoundationDB uses `version: v7.3` or `v7.4`, mapped to `7.3.63` and
`7.4.1`. Migration 60 carries the old `cluster.version` over to its
release line, records releases needing attention and preserves the
legacy value while the old chart can still render it.
- The FoundationDB chart refuses a render when `cluster.version` and
`version` name different release lines. This also covers variants
without platform migrations.
- OpenSearch and FoundationDB get version-map update scripts. They
retain the current default while it remains supported.

Application types and schemas are still generated from the charts.

### Upgrade notes

Stored image overrides are accepted without new admission warnings and
ignored by the charts. On the first reconcile, OpenSearch nodes move to
the image for their selected version, and Kubernetes components whose
images changed restart. Check custom OpenSearch images before upgrading:
if one runs a newer version than the chart's map, switching back can
leave nodes unable to read their data. Air-gapped installations need
`docker.io` and `ghcr.io` mirrored on management-cluster nodes.

FoundationDB preserves the release line, not an arbitrary patch: 7.3.x
moves to 7.3.63 and 7.4.x to 7.4.1. Check these versions before
upgrading. A cluster on 7.1 needs an upgrade to 7.3 first. On variants
without migrations, set `version` to the line the cluster runs. To
change release lines later, remove the leftover `cluster.version` as
directed by the chart error.

Migration 60 records affected releases in the
`cozystack.io/migration-60-needs-attention` annotation on the
`cozystack-version` ConfigMap. Dry-run reports patch changes without
patching, annotating or advancing the migration stamp. Migration 56 no
longer carries the removed kubectl override into adopted node pools.

An explicit OpenSearch `version: null` now fails rendering instead of
falling back to `v2`.

### Validation

- `make unit-tests` passed, including migrations 56 and 60.
- Four-chart regression run: 750 tests passed.
- API Create/Update, admission, defaulting and existing warning
regression tests passed. `cmd/`, `internal/` and `pkg/` match main.
- Shell test lanes passed with `/bin/sh` as dash throughout the process
tree.
- `make generate manifests` passed with no generated drift, using
cozyvalues-gen v1.7.0.

E2E passed on e694b05b4.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
existing generated parameter tables are refreshed by the release docs
job.
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: [schema
updates](https://github.com/cozystack/terraform-provider-cozystack/issues/74).
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The provider follow-up removes the Kubernetes and KubernetesNodes image
blocks and adds FoundationDB version selection. No other downstream
follow-up is needed for these fields.

### Release note

```release-note
fix(kubernetes, opensearch, foundationdb)!: Remove tenant image overrides from Kubernetes, KubernetesNodes and OpenSearch. Existing overrides switch to platform images on reconciliation, which can roll OpenSearch nodes and restart Kubernetes components. Air-gapped installations must mirror the upstream registries on management-cluster nodes. FoundationDB replaces cluster.version with version (v7.3 or v7.4); migration 60 preserves the release line and records patch changes. Check running image versions before upgrading. Variants without migrations must set the FoundationDB version before upgrading, and the chart refuses mismatched legacy release lines. OpenSearch rejects an explicit version: null.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* FoundationDB now supports selecting version lines `v7.3` and `v7.4`,
with `v7.3` as the default.
* Added automatic migration of existing FoundationDB version settings,
with warnings for settings that need attention.

* **Changes**
* Removed per-application image overrides for Kubernetes, Kubernetes
Nodes, and OpenSearch. These charts now use their bundled images, which
can still be routed through the configured image registry.
* OpenSearch now reports an error when its version is missing or
unsupported.
* Existing FoundationDB settings with conflicting version lines are
reported for review.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-26 13:00:10 +03:00
Mattia Eleuteri dc94660a44 fix(postgres): render the declared databases in a stable order
keys returns a map's keys in Go's randomized iteration order, so a
release with several databases rendered DEFINED_DBS in a different
order on each upgrade. The init-script Secret, and the checksum the
init Job carries, then changed on upgrades that left the databases
untouched, which makes a no-op upgrade look like a real change in the
release history.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-24 17:05:20 +02:00
myasnikovdaniil 55aad6f914 fix(tcp-balancer): cap HAProxy fds and threads by the pod's own limits (#5005)
## What this PR does

On a v1.6.5 cluster a TCPBalancer created with the chart defaults
(`t1.nano`, limits 250m CPU and 128Mi memory, `haproxy:3.4.4`)
crash-looped. Both pods were OOMKilled with a memory limit of 134217728,
7 restarts each in about 15 minutes, and HAProxy logged `[NOTICE] (8) :
Automatically setting global.maxconn to 524126.` On another cluster the
same chart picked maxconn 524251 and the pods kept running.

`haproxy.cfg` has no `global` section, so HAProxy sizes itself from the
node. maxconn comes from the nofile limit capped at the built-in
`fd-hard-limit` of 1048576 (`DEFAULT_MAXFD`), and the thread count is
the number of CPUs the process may run on, whatever its CPU limit is.
Both are paid for at startup, before any traffic, and the thread part
grows with the node.

This PR adds a `global` section sized from the replica's own limits,
read through the same `cozy-lib.resources.defaultingSanitize` helper the
Deployment uses, so `resources` and `resourcesPreset` resolve the same
way. For `t1.nano` it renders

```
global
    fd-hard-limit 4096
    thread-hard-limit 1
```

`fd-hard-limit` is 32 per MiB of the memory limit and HAProxy derives
maxconn from it, 2035 for `t1.nano`, which is about 64 KiB per
connection. For comparison `haproxy -m 128` picks maxconn 3600 on 3.4.4,
about 35 KiB per connection, and the difference is left for the process
itself and for the kernel socket buffers the cgroup also charges. An
explicit `maxconn` would read simpler, but `strict-limits` is on by
default and HAProxy refuses to start when it cannot raise nofile to
about twice that value. With 1024 soft and 4096 hard nofile `maxconn
4096` exits with `Cannot raise FD limit to 8251`, while `fd-hard-limit
8192` starts with maxconn 2018. `thread-hard-limit` is the CPU limit
rounded up, at least 1. It caps the detected thread count and never
forces more threads than the node has.

Reproduced with Docker Desktop on arm64 (12 vCPU, kernel 6.12) and the
same image, running the rendered config the way the Deployment mounts
it. A larger node is stood in for by adding `nbthread 96` or `nbthread
128` to the `global` section. Memory is `memory.peak` of the container
cgroup.

```sh
cd packages/apps/tcp-balancer && mkdir -p /tmp/cfg
helm template tb . --show-only templates/configmap.yaml | yq '.data["haproxy.cfg"]' > /tmp/cfg/haproxy.cfg
docker run --rm --memory 128m --memory-swap 128m --ulimit nofile=1048576:1048576 -v /tmp/cfg:/usr/local/etc/haproxy:ro haproxy:3.4.4
```

| run | main | this PR |
| --- | --- | --- |
| idle, 12 threads | maxconn 524258, peak 89.6 MiB | maxconn 2035, 1
thread, peak 18.7 MiB |
| idle, `nbthread 96` | maxconn 524127, worker exited with code 137,
OOMKilled | 1 thread, peak 20.8 MiB |
| idle, `nbthread 128` | worker exited with code 137, OOMKilled | 1
thread, peak 22.9 MiB |
| 2035 clients echoing 16 KiB through a backend for 25 s, 12 threads |
worker exited with code 137 about a second after the clients started |
running, peak 83.7 MiB |

One case this PR does not cover is clients and servers that both stop
reading. They fill kernel socket buffers that the cgroup charges, and
with this PR the 128Mi container was still OOMKilled after 1225 such
connections, which maxconn does not bound.

`helm unittest .` passes 13 of 13 with helm-unittest 1.1.2 as CI pins
it. Against the template on main the four new cases fail and the nine
existing ones pass, and each of six template mutations fails at least
one case. `haproxy -c` accepts the render for `t1.nano`, `t1.4xlarge`,
`m1.4xlarge`, a zero CPU limit, the whitelist, `kubernetesAndTalos` and
`tcp-with-proxy` inputs. A full `helm template` differs from main only
in the `global` section and the `checksum/config` annotation, so the
pods roll once on upgrade. I have not run the change on a live cluster.

The template is the same on release-1.6, and that backport is a separate
PR.

### Screenshots

Not a UI change.

### Downstream repositories

One chart template and its unittest suite in
`packages/apps/tcp-balancer`. No values key, schema, default, README or
package change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tcp-balancer): HAProxy now takes its fd and thread caps from the replica's memory and CPU limits instead of the node, so a t1.nano balancer is no longer OOMKilled on nodes with many CPUs. maxconn follows the memory limit at about 16 connections per MiB, 2035 on t1.nano, and the pods roll once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- TCP Balancer now automatically configures HAProxy file-descriptor and
thread limits based on the selected or explicitly defined resource
settings.
- CPU-based thread limits are rounded up and always maintain a minimum
of one thread.
- Memory-based file-descriptor limits scale automatically with the
configured memory capacity.

- **Tests**
- Added coverage for default, preset-based, explicit resource, rounding,
and minimum-limit scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-23 23:53:39 +05:00
myasnikovdaniil 614847976f test(harbor): pin the resources of the nginx proxy and the portal (#5025)
## What this PR does

#4095 gave the Harbor nginx proxy the `t1.small` preset and moved the
portal from `t1.nano` to `t1.micro`, so neither falls back to the
LimitRange default of a tenant namespace, 250m CPU and 128Mi. It came
without a test. This adds a helm-unittest suite that pins both presets
in the nested HelmRelease and checks that they stay the same when the
other components are resized, so dropping them or tying them to another
component's values fails the suite.

Both containers run nginx with `worker_processes auto`, which starts one
worker per CPU of the node whatever the container's CPU limit is. With
the proxy config the chart renders and `goharbor/nginx-photon:v2.15.1`,
96 workers idle at about 205 MiB, so on a node with many CPUs the proxy
is OOMKilled at 128Mi before it serves a request. That is what happened
on a v1.6.5 cluster, where both nginx pods of a default Harbor restarted
5 and 6 times within minutes, because `release-1.6` never got #4095. The
backport, with this suite and the measurements, is a separate PR.

`helm unittest .` in `packages/apps/harbor` passes 47 of 47 with
helm-unittest 1.1.2. With the #4095 hunk reverted the three new cases
fail and the 44 existing ones pass. Each of seven template mutations
fails at least one new case. They drop the nginx block, give nginx
`t1.micro` or the portal `t1.nano`, tie nginx to the core values or the
portal to the jobservice values, misspell the `resources` key under
nginx, and make core ignore its preset, which the resize case checks so
that it cannot pass on overrides that never reach the release.

### Screenshots

Not a UI change.

### Downstream repositories

One new helm-unittest suite in `packages/apps/harbor/tests`. No
template, values key, schema, default, README or package change, so no
trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* Added Helm coverage verifying default resource settings for Harbor’s
nginx and portal components.
* Added coverage confirming nginx and portal memory allocations remain
consistent when related service resources are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-23 22:40:28 +05:00
Myasnikov Daniil 5265ee662c fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates
helm-controller waits on every object of a release with the kstatus
poller on install and on every upgrade. kstatus reads a TenantGateway at
Ready=False/ListenersNotReady as InProgress, and a TenantGateway sits
there whenever one listener certificate is pending, which depends on
ACME or DNS and, under HTTP-01, on routes that other releases own. After
a platform upgrade the gateway release therefore times out, retries
without end under RetryOnFailure, holds a helm-controller worker
meanwhile, and fails the tenant release that waits on it. It used to
pass because health is judged only inside a Helm action: the release
was installed while the TenantGateway had no status yet or was still
Ready, and the listeners whose certificates do not issue came later
with other releases' routes.

An always-true expression replaces kstatus for TenantGateway and keeps
the wait on. Flux checks status.observedGeneration against the
generation before it evaluates any expression, so the release still
waits until the controller has observed the new spec, but not for
listener or certificate health. The poller strategy is set explicitly
because expressions are only evaluated under it.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 14:34:55 +00:00
Myasnikov Daniil 639ede20be fix(api): leave FoundationDB version checks in the chart
The chart still reads cluster.version to guard release-line changes,
so calling it ignored in an admission warning contradicts the render
error. Keep that decision in the chart, which can explain the version
mismatch, and limit removed-field warnings to inert values.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 19:14:17 +05:00
Andrei Kvapil b43196e549 fix(kubernetes): pin the Proxmox cloud-config builders to the bundled image
The Proxmox substrate added two init containers that build the cloud
config from the hypervisor token Secret, and both still took their
image from images.waitForKubeconfig. With that key gone the render
failed on a nil map; with it set, a tenant chose the image that reads
the token. They now run the bundled busybox through cozy-lib.image, like
the wait-for-kubeconfig container next to them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:17 +05:00
Andrei Kvapil 9775628cc8 fix(foundationdb): refuse a render that would move the release line
Migration 60 carries cluster.version over to version, but platform
migrations run only on the variants that enable them. On any other the
new chart ignored cluster.version and rendered the v7.3 default, so a
7.4 cluster was downgraded and a 7.1 one moved onto a line nothing had
upgraded it to.

The render now fails when a stored cluster.version names another
release line than version selects, and says how to proceed, so the
release stays on the revision it runs until someone picks the line.
Where the migration ran the two agree and nothing changes.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00