admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-10-25 04:27:04 +09:00
Code Issues Packages Projects Releases Wiki Activity
7,595 Commits 523 Branches 291 Tags
feat/postgres-credentials
Commit Graph
7595 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Timofei Larkin 6dfa5c71e2 feat(platform): migrate Postgres accounts to platform-minted credentials 
The converted postgres chart reads each account's verifier from an
account Secret the API writes, and nothing seeds those Secrets for the
applications that exist before the chart switches: the API seeds only
on a write of the application, and the old chart keeps serving the old
passwords from <release>-credentials until the switch. Without a
migration every existing account would stop logging in at the switch,
since the operator leaves a role whose Secret is missing as it is but
the chart no longer applies the old password either.

Migration 60 runs in the pre-upgrade hook, before any component chart
changes version, and writes the account Secrets with the verifiers of
the chart-generated passwords, origin Migrated, so clients keep logging
in at every point, and records the chart version of each release. A
Job the platform release creates, not a hook so a long wait does not
hold the upgrade and so it can be run again by hand, waits per release
until the converted chart is Ready, removes any users.*.password left
in the values and deletes <release>-credentials, which no platform
account of postgres shares. It exits non-zero while a release is left
out. The verifier is computed by the API's own format function, built
into a CLI the migrations image carries, so a migrated release and a
mint store the same form; the image is built from the repository root
for that reason.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-10-14 12:50:05 +02:00
Timofei Larkin 97a3eaa82a feat(postgres)!: mint tenant passwords through the credentials API 
The chart generated every user's password at render time into
<release>-credentials, which tenants could read again at any time and
which Helm history kept for MaxHistory revisions, so a deployment whose
policy requires a credential shown once could not run PostgreSQL here.

The accounts move under CloudNativePG's managed roles, each pointing at
the account Secret the Cozystack API seeds, so the operator creates the
role with the SCRAM verifier the Secret holds and applies a changed
Secret live, and a password set inside PostgreSQL is reverted. Every
role is rendered as the init-script created it, because the operator
reverts an attribute that differs, revokes a parent role missing from
inRoles and clears a comment the spec leaves unset, while the
init-script still finds the roles to drop by that comment and by
membership in the marker. The init-script stops creating, passwording
and altering accounts, where it would be a second writer, and keeps the
two things the operator has no field for: the marker membership
narrowed to a record, and createrole_self_grant.

The refusal to adopt a role the chart did not mark moves into a
pre-upgrade hook, because the operator adopts a managed role within
seconds of the spec reaching it and the post-upgrade init-job would
find it adopted already. The definition declares the credentials block,
so the API seeds, serves and mints for the kind, and no longer selects
the credentials Secret, which the chart no longer renders or grants.

BREAKING CHANGE: PostgreSQL passwords are no longer readable from
<release>-credentials. A password is issued once, by a mint on the
account's Credential in core.cozystack.io, and replaced by the next
mint; deleting a Postgres application whose release is suspended is
refused. The conversion needs a platform variant with migrations
enabled.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-10-14 12:50:05 +02:00
Timofei Larkin 069054b697 docs(api): say why two password formats hash without a salt 
CodeQL flags the SHA-1 and SHA-256 of a password as weak hashing of
sensitive data. Both are the forms the engines store and check against
and accept in no other form, and the design relies on the 32-character
random password rather than on the digest for strength, so the alerts
are dismissed with that reason and the code says it where the hashing
happens.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-10-14 12:50:02 +02:00
Timofei Larkin 108b51df3d feat(api): serve credentials with mint and revoke, and seed accounts 
The Credential view lists the account Secrets of the namespace by label
and skips any no live HelmRelease owns, so no kind is served until its
definition carries a credentials block. A mint replaces the stored value
in one conditional write keyed on the Secret's resourceVersion and
returns the plaintext once; a revoke discards it. Both record an Event
on the application, which needs create on events.

The application registry seeds an account Secret per declared account
before it writes the HelmRelease, so the chart can reference them from
the first render, prunes the Secrets of removed accounts after the
write, and refuses to delete an application whose release is suspended,
since Flux skips the uninstall and would leave the database running
with the Secrets gone. The ApplicationDefinition reconciler computes
the per-account valuesFrom entries with the function the API uses, so a
release that predates the block gets them on its next reconcile. Schema
defaulting moves to a package both can import.

A definition declaring a format or an account source that is not built
leaves its kind unserved rather than served with passwords the API
cannot issue. Tenant roles gain get and list on credentials at view and
create on mint and revoke at admin and for the tenant ServiceAccount.
Attribution comes from the kube-apiserver audit log, so the operations
docs get a policy for distributions that ship none, and
monitoring-agents a value for an extra fluent-bit output so a SIEM can
receive the log without dropping the VictoriaLogs outputs.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-10-14 12:29:46 +02:00
Timofei Larkin 0bcded822e feat(api): add Credential kinds and the credentials block 
Cozystack generates managed application passwords at Helm render time
and keeps them in a Secret tenants can read again at any time, which a
deployment whose policy requires a credential shown once cannot accept.
The accepted design moves generation into the API: an
ApplicationDefinition declares its accounts and the stored form of a
password in a credentials block, and each account gets a
core.cozystack.io Credential whose mint and revoke subresources take a
TokenRequest-shaped CredentialRequest.

This adds the types, the built-in password formats, which the migration
CLI shares so a converted release and a mint store the same form, and
the account helpers that name the Secrets and read their record.

Design: https://github.com/cozystack/community/blob/main/design-proposals/user-secrets-api/README.md

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-10-14 12:29:46 +02:00
Aleksei Sviridkin e05c56524e fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427) 
## What this PR does

Closes the uncovered VM backup path from #3236.
`hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the
old `backup.bats`, written against the retired API (`kind:
VirtualMachine`, `BackupJob.storageRef`/`strategyRef`,
`velero-strategy-default`), so it guarded nothing. This removes it,
fixes the in-place restore of a standalone VMDisk in the backup
controller, and adds two real round-trips in the `vminstance` suite that
drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented
demo and the automated test stay in step.

### Controller: standalone VMDisk in-place restore

`collectUnderlyingResources` in
`internal/backupcontroller/velerostrategy_controller.go` returned
nothing for any kind but VMInstance, so `prepareForRestore` did nothing
for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete,
no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux
and CDI over the PVC name the data mover was repopulating. A standalone
VMDisk now reports its own disk as its underlying resource, so the
restore preparation already used for VMInstance disks suspends the
`vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the
live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests
cover both the VMDisk and the VMInstance branch.

A VMInstance is halted before its disks are touched, but a VMDisk target
is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a
disk that a running VM mounts would leave the original `Terminating`
behind `pvc-protection`; Velero's PVC restore action skips a PVC that
still exists, so the RestoreJob would report `Succeeded` having moved
nothing, and the VM would lose its disk on the next restart. An in-place
VMDisk restore therefore fails before changing anything when a Pod that
has not terminated mounts the disk's PVC. The message names the Pod and
tells the user to stop the VM or restore the VMInstance instead.
`TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod
(refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a
Pod on another disk, and a VMInstance target whose own launcher Pod must
not refuse it.

A VMDisk Backup taken before the disk was recorded as its underlying
resource has none, so the disk is derived from the application name
(`vm-disk-<name>`): such a restore is prepared and checked for a
mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance
concern: a VMDisk strategy already selects its disk, and Velero rejects
a Backup or Restore carrying both `labelSelector` and
`orLabelSelectors`, so appending them for a VMDisk would break a
strategy written with `labelSelector`.

When an in-place restore fails after preparation, nothing is rolled
back. Resuming the disk's HelmRelease while its PVC is renamed and its
DataVolume deleted would have CDI import a fresh disk under the original
name, which looks healthy and hides that the data sits in the `-orig-`
PVC. The failure message and a Warning event instead name the
HelmReleases preparation suspended and the `-orig-` PVCs holding the
original data, read back from the cluster so that a failure before a
step ran does not name what it would have left, and say that a
HelmRelease Velero already restored is resumed, so the disk has to be
checked; `examples/backups/vmdisk/README.md` describes the recovery.
Tests cover both selector paths, the pre-upgrade Backup, a failed Pod
list (the restore requeues and suspends nothing), and all three failures
after preparation: a `PartiallyFailed` Velero Restore, several Restores
owning the job, and a Restore that cannot be created.

Because preparation takes the target apart before Velero validates the
Restore, the RestoreJob is refused before preparation, with nothing
changed, when the Velero Backup is gone (expired or deleted) or not
`Completed`. This is the same single check #4672 added, now run on every
pass behind the preparation failure path, so a Backup that expires after
preparation started names the leftovers too. A Velero Restore that ends
in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's
validation errors and the leftovers in the message, instead of polling
forever. A HelmRelease suspend or a VM halt that does not happen now
stops preparation (requeue) rather than letting the rename and
DataVolume delete run under a live release or VM, and a VMInstance's
disks are renamed only once no Pod mounts them. A VMDisk held only by a
Pod already being deleted is waited for rather than refused, for up to
30 minutes after the RestoreJob started, so a Pod stuck terminating on a
lost node fails the job instead of keeping it `Running`. Each of these
is pinned by a test that turns red when the guard is removed.

Between renaming a kept disk's PVC and deleting its DataVolume, CDI can
recreate an empty PVC under the original name for the still-present
DataVolume. Velero then skips that PVC as already present ("PVC already
exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`,
and the VM boots a blank disk. The round-trip's new DataDownload check
caught this on a VMInstance restore, where the uid, label and `-orig-`
checks had all passed. Once the DataVolume is gone (the wait #4672
added), preparation now deletes a PVC found under the original name
(read uncached) before the Velero Restore is created;
`TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the
recreation inside the DataVolume delete.

Preparation spans several reconciles, so the RestoreJob records a
`PreparationStarted` condition before it first changes the target; if
recording it fails, preparation retries without changing anything, so a
later failure cannot report leftovers that were never made. A refusal on
a later pass (the Velero Backup expired, a VM started on the disk) then
names what preparation left behind instead of saying nothing was
changed, and a preparation still waiting 30 minutes after it started
fails the same way instead of polling forever. The vmdisk script starts
its consumer Pod only on a PVC that carries the restore label, so it
cannot pin a PVC CDI recreated in the rename window.

Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches
it as well. Its DataVolume is deleted with orphan propagation, and the
bound PVC is deleted and waited for before the Restore (the path #4672
added for VMInstance disks). Left in place, Velero would skip the PVC
and the CDI webhook would refuse the DataVolume the resumed HelmRelease
creates, leaving the disk's HelmRelease stuck.
`TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it.

### `vminstance-2-backup-roundtrip` (un-gated in CI)

A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the
current API end to end: Velero strategy, BackupClass, a booted
VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each
waited to `Succeeded` and failing fast on `Failed`. The VM must come
back `Running`, and its disk PVC must have been created by the restore:
a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the
`vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The
label alone does not prove the data moved, because with
`existingResourcePolicy: update` Velero also stamps it on a PVC it only
patched, and the patch path never reaches the data mover. An in-guest
signal is not available on this platform: the KubeVirt CR disables
serial-console logging cluster-wide and cirros ships no
qemu-guest-agent.

### `vmdisk-2-backup-roundtrip` (un-gated in CI)

The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore
it in place relying on the controller fix above, and require the same
three signals (new uid, restore label, `-orig-` PVC), a `Completed`
DataDownload for the restore that wrote bytes, and a Ready HelmRelease.
On a `WaitForFirstConsumer` StorageClass, such as the `local` class the
CI lane uses, the Velero data mover waits until a scheduled Pod has
selected a node for the restored PVC (`WaitPVCConsumed` in Velero's
generic restore exposer). A VMInstance gets that from its virt-launcher
Pod; a detached disk has none, so its `RestoreJob` stays `Running` until
the 30m data-mover prepare timeout fails it. The script therefore starts
a restricted-profile `pause` Pod on the new PVC once Velero has created
it, standing in for the VM that would attach the disk, and removes it
after the restore. `examples/backups/vmdisk/README.md` documents this
for operators restoring a detached disk.

### Harness details

- The disk `storageClass` is an override (default `replicated`). The
Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the
DRBD-off PR gate).
- Each harness stands up a throwaway in-cluster BSL
(`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333`
with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236
fixes), reusing the platform bucket and `cozy-backups-creds` under a
distinct prefix, because the platform `cozy-default` BSL derives the
external S3 ingress endpoint, unroutable in the CI cluster.
- The script waits are bounded so that their worst case stays under the
40m op timeout (38m for the VMDisk step), and a hung wait trips the
script's own named timeout rather than a bare `signal: killed`.
- On failure, `catch` also describes the Velero `Restore`,
`DataDownload` and `DataUpload` objects in `cozy-velero`.
- `skipDelete` plus a `finally` that resumes the suspended HelmReleases,
reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back
to `Delete` first), and drains the Velero backups on the throwaway BSL,
branching on kubectl's exit status, before deleting it. When the drain
is not confirmed, the BSL is kept and the step fails, so the Velero
Backups are not orphaned behind a green step.
- HelmRelease waits use the shared `wait_hr_ready`, which fails fast on
`Stalled`. After the VMDisk restore the script first waits for
`observedGeneration` to reach the restored spec's `generation`, because
the `Ready=True` from before the suspend would otherwise satisfy the
wait at once.

The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are
non-interactive, the VM is waited to boot before the backup,
`volumeSnapshotLocations` is dropped from the demo strategies
(`snapshotMoveData` routes data through the BSL), and
`BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh`
maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite,
except `07-restore-to-copy.sh`, which the suite skips
(`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A
`packages/system/velero` change selects the `vminstance` suite as well,
since these are the only Tests that run a Velero backup and restore.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the diff
(backup controller restore preparation for VMDisk, shell scripts under
`examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a
deleted stale suite, one doc line): no `packages/apps|extra` add/rename,
no `values.schema.json`/enum/default changes, no
CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The
controller change alters what a VMDisk `RestoreJob` does in the cluster,
not any API surface a downstream repository consumes.
`cozystack/examples` holds Terraform for bare nodes, unrelated to the
in-repo `examples/` tree.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI.
```
 2026-10-13 23:40:02 +03:00
Aleksei Sviridkin 588d8292e7 fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#4819) 
## What this PR does

Large objects served through a Bucket's S3 endpoint stop tens of
megabytes into the download (cozystack/cozystack#3782). The cause is not
TLS: the SeaweedFS S3 gateway is OOM-killed mid-response. Its read-ahead
for a sequential GET is meant to stay four chunks in front of the
reader, but SeaweedFS counts only newly started chunk downloads toward
that window and re-runs the prefetch after every 256 KiB read, so the
gateway pulls up to about a gigabyte of the object into memory and
exceeds the 512Mi limit the S3 deployment gets from its resources
preset. The client sees `unexpected eof` over HTTP/1.1 and
`INTERNAL_ERROR` over HTTP/2, which is what the CDI importer reported.
Upgrading does not help yet: 4.31, 4.45 and the latest release 4.48 all
carry the defect.

The defect is reported upstream in seaweedfs/seaweedfs#11672, and the
fix with its regression test is merged there in
seaweedfs/seaweedfs#11674, after 4.48. Until a SeaweedFS release
carrying it is vendored, this PR builds SeaweedFS in-tree the way #3343
builds cilium-operator: `images/seaweedfs` downloads the 4.45 source
archive and checks it against a pinned checksum, applies
`patches/reader-cache-prefetch-window.diff`, runs the tests of the
packages the patch touches (including the regression test the patch
adds) once, compiles `weed` with the Go toolchain upstream's own builder
uses for the tag, and lays it over the upstream
`chrislusf/seaweedfs:4.45` image pinned by digest, so everything else in
the image stays the upstream build. `weed version` reports
`cozystack-patched` in the commit field. All SeaweedFS components run
the in-tree image.

`make update` re-pins the Dockerfile from the vendored chart: the
version, the source checksum, the upstream image digest and the builder
toolchain. It fails when the patch no longer applies, which is how the
first release carrying the upstream fix will show that the patch is
ready to drop. Renovate is disabled for that Dockerfile, since it would
move those pins one at a time, and whitespace checks are off for the
patch.

Tests:

- `tests/image_test.yaml` pins every component (master, volume, filer,
S3) to the in-tree image with a digest.
- `make test-image-version` fails when the Dockerfile's version drifts
from the vendored chart's appVersion.
- `make update-image`, which `make update` runs, was checked to be
idempotent on the current chart, to restore deliberately corrupted pins,
and to fail with a clear message when the patch context no longer
matches.
- `hack/build-matrix_test.bats` now expects seaweedfs to be a build unit
next to objectstorage-controller.
- The Go regression test inside the patch fails on unpatched 4.45
(read-ahead reaches 60 chunks past the reader) and passes with the
patch; the image build runs it.

Verified on a test cluster against an 840 MiB object, with every
SeaweedFS component running the image this package builds and the S3
gateway at the production limits (512Mi, 1 CPU). Unpatched 4.31, 4.45
and 4.48 gateways were OOM-killed on every full GET. The patched gateway
served nine full GETs (six sequential over HTTP/1.1 and HTTP/2, three in
parallel) with matching checksums and no restarts; it read from the
volume servers about as many bytes as it sent, its live heap after a GET
was about 40 MiB, and three parallel GETs peaked at about 140 MiB. The
image this PR pins reports `4.45 cozystack-patched` and was rerun
through the same checks.

Drop the patch, and the in-tree build with it, once a SeaweedFS release
carrying the upstream fix is vendored.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(seaweedfs): stop the S3 gateway from being OOM-killed while serving large objects, which cut downloads (including CDI imports from a Bucket URL) off tens of megabytes in. SeaweedFS is now built in-tree with a patch that bounds the gateway's read-ahead.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* SeaweedFS now runs a Cozystack-built image based on version 4.45,
pinned to a specific image digest.

* **Bug Fixes**
* Sequential small reads now keep read-ahead within the configured
prefetch window, preventing it from extending unexpectedly.

* **Build & Maintenance**
* SeaweedFS image updates are aligned with the chart version and verify
the source archive, builder image, and patches before building.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-13 20:08:11 +03:00
Andrey Kolkov 58b4343e5a ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite 
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).

The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-10-13 18:07:35 +04:00
Andrey Kolkov 9cc3a9e459 test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips 
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.

Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
  up via BackupClass + BackupJob, restores in place, and asserts the VM returns
  Running off a disk PVC the restore created: a new uid, Velero's restore-name
  label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
  enough, because existingResourcePolicy: update also stamps it on a PVC Velero
  only patched, which moves no data. A guest-side boot proof is unavailable: the
  KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
  place with the same three assertions. The controller prepares the target
  (suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
  class the Velero data mover waits for a Pod to select a node for the restored
  PVC, which a detached disk never gets, so the script starts a pause Pod on the
  new PVC, standing in for the VM that would attach it.

The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-10-13 18:05:45 +04:00
Andrey Kolkov 9af1c0f902 fix(backups): prepare standalone VMDisk restore targets before the data mover runs 
collectUnderlyingResources returned nothing for a non-VMInstance kind, so
prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and
no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the
restored HelmRelease reinstalled and CDI recreated the DataVolume over the same
PVC name the data mover was repopulating, wedging the RestoreJob.

Report a standalone VMDisk's own disk as its underlying resource, so the existing
(VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease,
deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before
the data mover recreates and repopulates it. No VM, so no IP/MAC is collected.

A VMInstance is halted before its disks are touched; a VMDisk target is not,
and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM
mounts leaves the original Terminating behind pvc-protection, Velero skips a
PVC that still exists, and the RestoreJob would report Succeeded having moved
nothing while the VM loses its disk on the next restart. An in-place VMDisk
restore now fails before changing anything when a Pod that has not terminated
mounts the disk's PVC, naming the Pod and telling the user to stop the VM or
restore the VMInstance instead.

A VMDisk Backup taken before the disk was recorded as its underlying
resource has none, so its disk is now derived from the application name
and such a restore is prepared and checked for a mounting Pod as well.

The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk
strategy already selects its disk, and Velero rejects a Backup or
Restore that carries both labelSelector and orLabelSelectors, so
appending them for a VMDisk would break a strategy written with
labelSelector.

When an in-place restore fails after preparation, nothing is rolled
back: resuming the disk's HelmRelease while its PVC is renamed and its
DataVolume deleted would have CDI import a fresh disk under the original
name, which looks healthy and hides that the data sits in the -orig-
PVC. The failure message and a Warning event now name the suspended
HelmReleases and the -orig- PVCs so the user can recover them, read back
from the cluster, so a failure before a step ran does not name what it
would have left.

Preparation takes the target apart before Velero validates the Restore,
so it is refused up front when the Velero Backup is gone or not
Completed. That check now runs on every pass behind the same failure
path as the rest of preparation, and a FailedValidation Restore fails
the RestoreJob like a Failed one instead of leaving it Running. A
HelmRelease suspend or a VM halt that did not happen now stops
preparation instead of renaming and deleting under a running release or
VM, and a VMInstance's disks are renamed only once its launcher Pod has
let go of them. A VMDisk held only by a Pod already being deleted is
waited for rather than refused, for
up to 30 minutes after the RestoreJob started.

Between renaming a kept disk's PVC and deleting its DataVolume, CDI can
recreate an empty PVC under the original name. Velero then skips the
PVC as already present, the RestoreJob reports Succeeded and the VM
boots a blank disk. Once the DataVolume is gone, preparation now
deletes a PVC found under the original name, read uncached, before the
Velero Restore is created.

Preparation spans several reconciles, so the RestoreJob records a
PreparationStarted condition before it first changes the target, and
retries without changing anything when recording it fails. A
refusal on a later pass (the Velero Backup expired, a VM started on the
disk) then names what preparation left behind instead of claiming that
nothing changed, and a preparation still waiting 30 minutes after it
started fails the same way instead of polling forever.

Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it
too: its DataVolume is deleted with orphan propagation and the bound PVC
is deleted before the Restore, as for a VMInstance disk. Left in place,
Velero would skip it and CDI would refuse the DataVolume the resumed
HelmRelease creates.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-10-13 18:05:45 +04:00
Alexey Artamonov e8635f6e4b build(seaweedfs): stamp the re-pinned in-tree image 
The builder, the compile flags and the version stamp changed, so the
image built from them has a new digest, and values.yaml has to name it
for the components to run that build.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-10-13 16:22:45 +03:00
Alexey Artamonov ecdc043e63 build(seaweedfs): keep Renovate and git whitespace off the in-tree build 
Renovate would bump the builder, the base image or their digests one at
a time into a combination upstream never shipped, and the next `make
update` would silently put them back, so it leaves that Dockerfile to
the update recipe. The patches are unified diffs of Go sources, whose
context lines start with a marker before tabs and blank lines, which
git's whitespace checks would otherwise flag or rewrite.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-10-13 16:22:45 +03:00
Alexey Artamonov 5592a3f65d build(seaweedfs): re-pin the in-tree build from make update 
The in-tree build pinned its source, base image and toolchain by hand,
and nothing moved them when `make update` vendored a new chart, so a
bump would build one release's weed into another release's image, or
keep patching a release that already carries the fix.

Pin the source archive by checksum and the upstream image by digest as
build arguments, compile with the toolchain upstream's own builder uses
for the tag, and let `make update` re-pin all of them from the vendored
chart and fail when the patches no longer apply. The tests of the
patched packages now run once rather than once per target platform,
and `weed version` names the build.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-10-13 16:22:44 +03:00
Alexey Artamonov d2c6b85cac fix(seaweedfs): run the in-tree SeaweedFS image in every component 
With the image keys unset the vendored chart renders chrislusf/seaweedfs
at its appVersion, so the patched build from images/seaweedfs would be
built and never run, and the S3 gateway would keep being OOM-killed on
large GETs. Point every component at the in-tree image, pinned by
digest like every first-party reference, and hold that in a unit test
so a values change cannot quietly fall back to the upstream binary.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-10-13 16:22:44 +03:00
Alexey Artamonov 1913104bd3 build(seaweedfs): build SeaweedFS in-tree with a bounded read-ahead 
The S3 gateway is OOM-killed while serving a large object. SeaweedFS
starts read-ahead downloads after every read of a sequential stream and
counts only the downloads it newly starts toward its prefetch window,
so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to
about a gigabyte of the object into memory, past the 512Mi limit of
the S3 deployment. No SeaweedFS release carries a fix yet, so there is
no release to move to instead.

Build weed from the 4.45 sources with a patch that keeps the window at
the next prefetchCount chunks, and lay it over the upstream image of
the same tag so the rest of the image stays upstream's own. The image
build runs the tests of the packages the patch touches, including the
regression test the patch adds, because go build never compiles them.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-10-13 16:22:43 +03:00
Aleksei Sviridkin c0449eeabb feat(backups): back up Valkey with the Redis strategy (#4898) 
## What this PR does

Adds backup and restore for the managed Valkey app.

Valkey runs on the same spotahome RedisFailover operator as Redis
(`spec.engine: Valkey`), with the same sentinel topology, replication
protocol and RDB format. So instead of adding a new strategy kind, the
existing Redis strategy now serves both apps:

- **Driver.** `applicationRef.kind` may be `Redis` or `Valkey`. The
template context gets `.Release.Prefix`, which the driver derives from
the app's kind (`redis-` or `valkey-`). The chart uses that prefix for
every name it builds: the sentinel Service `rfs-<prefix><app>`, the auth
Secret `<prefix><app>-auth` and the CA Secret. In cleanup mode the
prefix comes from the Backup's `applicationRef`, so it works after the
app is gone. A kind with no known prefix fails the render instead of
silently producing an empty prefix. Error messages name the real kind.
- **Upgrade window.** The shipped template reads `{{ .Release.Prefix |
default "redis-" }}`. The strategy CR and the controller ship in one
release, and until the new controller takes the lease, the previous one
(which has no `Prefix` and serves only Redis) renders the new CR; a bare
reference would print `<no value>` into the Secret and sentinel names.
- **Same-kind restore only.** A restore whose target kind differs from
the backup's source kind fails before any Job is created. Replaying a
Redis dump into Valkey, or the reverse, is not something the strategy
can promise across engine releases.
- **Chart.** `cozy-default-redis` builds names from `.Release.Prefix`.
`cozy-default` routes `apps.cozystack.io/Valkey` to
`cozy-default-redis`; the DefaultObjectsGate already dedupes repeated
strategy refs. No new image: the `redis:8` base loads the RDB v11 that
both shipped Valkey versions write. The Valkey chart has no TLS option,
so the optional CA mount stays empty and the Job connects in plaintext.
- **CRD.** Only the `.Release.Prefix` description in the Redis strategy
CRD changes (regenerated with controller-gen v0.16.4; every other
strategy CRD regenerates byte-identical).
- **Docs.** `docs/operations/backup-classes.md` has the Valkey route and
a paragraph on how the shared strategy behaves.
- **E2E.** The Redis backup demo takes `APP_KIND=Valkey`, which switches
the app kind and the name prefix. Its BackupClass routes Valkey too. The
valkey suite gains `valkey-backup-roundtrip`, which seeds a marker,
backs it up, restores it in place and into a copy. In
`hack/select-e2e.sh`, a seaweedfs change now selects the valkey suite
(enforced by `select-e2e_test.bats`), an edit to the shared
`examples/backups/redis/` harness selects both `redis` and `valkey`
(pinned by a new test), and `hack/select-install.sh` keeps the backup
stack in the reduced install for the valkey suite (pinned in
`select-install_test.bats`).

### Validation

- `go test ./internal/backupcontroller/...` passes. New tests: a Valkey
BackupJob renders `--prefix=valkey-`; a Redis backup restored into a
Valkey target fails without creating a Job; the ref validator accepts
Valkey; a cleanup for a kind with no prefix is released without a Job.
The existing Redis render test now also asserts `redis-`.
- The `default "redis-"` fallback, run through the controller's own
template engine with no `Prefix` in the context, renders
`rfs-redis-cache`; the bare reference renders `rfs-<no value>cache`.
- `helm unittest` on backupstrategy-controller: 26/26, with
`cozy-default` at eleven routes. `helm lint` is clean.
- `hack/redis-backup-strategy.bats` executes the shipped script and adds
a Valkey case: it must query `rfs-valkey-<app>` and never `rfs-redis-`.
The fixture now also fails if any `{{ }}` placeholder survives
substitution. All 62 tests in `hack/select-e2e_test.bats` pass.
- Engine compatibility, checked locally with the strategy image
`redis:8-alpine@sha256:ba6e…` against `valkey/valkey` 8.1.8 and 7.2.13
using the shipped script's exact steps: `redis-cli --rdb` from a
password-protected Valkey master, load into the throwaway
`redis-server`, emit the RESP base AOF, `FLUSHALL`, then `--pipe` into a
fresh Valkey. On both versions every check held: the db0/db1/db3
boundaries, a stream with its consumer group, a key's TTL and the marker
all survived, and FLUSHALL wiped the stale target key.
- I have not run this on a live cluster. The new chainsaw round-trip is
what exercises it end to end.

Out of scope, and already true before this PR: the website's backup
pages in `docs/next` don't list Redis, RabbitMQ or Kafka yet, so none of
the triggers in the downstream map fire here.

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(backups): Valkey apps can now be backed up and restored. The platform `cozy-default` BackupClass routes `apps.cozystack.io/Valkey` to the existing `cozy-default-redis` strategy, which dumps an RDB snapshot to the system bucket and restores it in place or into another Valkey app. A Valkey backup restores only into a Valkey app.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for backing up and restoring Valkey applications using
the existing Redis backup strategy.
* Updated the backup walkthrough to support both Redis and Valkey, with
resource names selected for the chosen application type.

* **Bug Fixes**
* Prevented restores when the backup and target application types
differ.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-13 15:57:54 +03:00
Aleksei Sviridkin 7f454d810c chore: link the FOSSA license scan for CLOMonitor (#4899) 
## What this PR does

Adds a root `.clomonitor.yml` that points CLOMonitor's
`license_scanning` check at the FOSSA project for this repository. This
is step 2 of #4758.

```yaml
licenseScanning:
  url: https://app.fossa.com/projects/git%2Bgithub.com%2Fcozystack%2Fcozystack
```

- **Why a metadata file rather than a badge.** The check passes on a
FOSSA or Snyk link in the README or on a URL in `.clomonitor.yml`
([check
docs](https://github.com/cncf/clomonitor/blob/main/docs/checks.md#license-scanning)).
#4222 trimmed the README badge row, so this uses the file.
- **A correction to #4758.** The key is `licenseScanning`, in camelCase.
CLOMonitor deserialises the file with serde `rename_all = "camelCase"`
([`metadata.rs`](https://github.com/cncf/clomonitor/blob/main/clomonitor-core/src/linter/metadata.rs),
[template](https://github.com/cncf/clomonitor/blob/main/docs/metadata/.clomonitor.yml)).
The snake_case `license_scanning` I proposed in the issue would have
been ignored silently.
- **Precondition from #4758 is met.** The FOSSA project is now analysed
at a current `main` revision (`last_analyzed_revision` = `3a463609e`,
2026-10-12, moved past `bdc3525c9`). It has 0 unresolved licensing
issues, and its update hook is the GitHub integration.

Checked: the file parses to exactly that one key, the URL matches
CLOMonitor's FOSSA regexp, and the project page is public (HTTP 200).

Expected effect: `license_scanning` passes on the next CLOMonitor scan,
worth about +4 to the project score.

This does not close #4758. Per @tym83, licensing findings from the fresh
scans stay to be triaged there.

### Downstream repositories

<!-- Walked the trigger map in docs/agents/contributing.md against the
diff: one new root metadata file read only by CLOMonitor; it touches no
package, values file, CRD, asset name, hack/ script or namespace. -->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added repository metadata describing license scanning and linking to
its results.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-13 14:45:26 +03:00
Andrey Kolkov 43b02bda30 test(e2e): run the Redis backup round-trip against Valkey 
The Redis backup demo now takes APP_KIND=Valkey, which switches the app
kind and the chart prefix the helpers derive sentinel and Secret names
from. The demo BackupClass routes Valkey to the same demo strategy.

The valkey suite gains valkey-backup-roundtrip, which drives that demo
end to end: seed a marker, back it up, restore it in place and into a
copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a
seaweedfs change now selects the valkey suite too, and since the
round-trip runs examples/backups/redis as its harness, an edit there
selects both the redis and the valkey suites. The reduced E2E install
keeps the backup stack for the valkey suite, as it does for the other
backup round-trip suites.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
 2026-10-13 12:34:24 +04:00
Andrey Kolkov 0c2f131192 feat(backups): back up Valkey with the Redis strategy 
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.

The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.

The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.

The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
 2026-10-13 12:34:24 +04:00
Marian Koreniuk d5c59aef22 chore: link the FOSSA license scan for CLOMonitor 
CLOMonitor's license_scanning check passes when the README links a FOSSA
or Snyk project, or when a root .clomonitor.yml names the results URL.
#4222 trimmed the README badge row, so use the metadata file.

The key is `licenseScanning` (camelCase). CLOMonitor deserialises the
file with serde `rename_all = "camelCase"`, so the snake_case
`license_scanning` proposed in #4758 would be ignored without an error.

The FOSSA project is analysed at a current main revision again
(3a463609e, 2026-10-12), with 0 unresolved licensing issues.

Refs: #4758

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
 2026-10-13 10:06:52 +02:00
Aleksei Sviridkin 3a463609e4 feat(bucket): add a ui flag to skip the s3manager web UI (#4881) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds a `ui` flag to the Bucket application. Every bucket currently ships
an s3manager Deployment, its Service, and an Ingress or HTTPRoute on
`<bucket>.<tenant host>`. Tenants that only use the S3 API pay for a pod
per bucket and for one DNS name and TLS certificate per bucket, none of
which they need.

- `packages/apps/bucket`: new `ui` value (default `true`), passed
through to the `bucket-system` HelmRelease. The dashboard Role drops its
Ingress rule when the UI is off, since an empty `resourceNames` would
widen it to every Ingress in the namespace.
- `packages/system/bucket`: the Deployment, Service, Ingress and
HTTPRoute render only when `ui` is true. With the UI off, the chart no
longer requires the tenant apex, as the UI hostname was its only
consumer there.
- Regenerated `values.schema.json`, `README.md`, `types.go` and the
`bucket-rd` ApplicationDefinition.

The default keeps existing buckets unchanged. Setting `ui: false` on an
existing bucket makes Helm delete the UI objects on the next upgrade;
the cert-manager Certificate for the UI host is owned by the Ingress
through ingress-shim and goes with it.

Tested with `helm unittest` on both charts: new suites cover the default
render, `ui: false` behind an Ingress and behind a Gateway, the
empty-apex case, the value passthrough and the dashboard Role. The
system-chart suite fails against the previous templates.

### Screenshots

Not applicable: no dashboard change, only a new field in the Bucket form
generated from the schema.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

Walked the trigger map against the diff. A new field in
`packages/apps/bucket/values.schema.json` reaches the Terraform provider
(schema, model, expand/flatten) and the website (the Bucket page is
hand-written). Both follow-ups are issues rather than PRs, since they
apply only once this PR is merged.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#750
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#71
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(bucket): add a `ui` flag to the Bucket application. Set it to `false` to skip the s3manager web UI and its Service, Ingress or HTTPRoute, DNS name and TLS certificate. Defaults to `true`, so existing buckets are unchanged.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a `ui` option for bucket deployments. It is enabled by default
and provides the S3 manager web UI at the tenant host.
* Set `ui` to `false` to run a bucket without its web UI or the
associated Service and Ingress or HTTPRoute.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:49:41 +03:00
Aleksei Sviridkin cfa9c5a5c7 fix(e2e): keep chainsaw asserts polling through transient API errors (#4867) 
## What this PR does

Chainsaw asserts in the e2e suites can fail long before their timeout.
Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other
API error. When etcd changes its leader, kube-apiserver answers the GET
with a 500, and whichever suite happens to be polling goes red. Between
2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached
chainsaw, which is 6 of the 23 assert failures in that window. Each one
has `etcdserver: leader changed` in the apiserver logs within 0.2 s of
the failure. Five of the six came after 10-09, and four of them were the
computeplane ingress-nginx assert.

The fix is kyverno/chainsaw#2761, which is still open and not in any
release. Until it is released, the e2e sandbox image builds chainsaw
v0.2.15 from source with that PR's commit applied unchanged, using the
same Go minor and build flags as the v0.2.15 release binary. The build
runs the tests of the patched packages. It also runs chainsaw against a
fake apiserver that answers 500: a short run of 500s must not fail the
assert, and an endless one must fail it at its timeout. Plain v0.2.15
fails both cases after the first GET.

The Dockerfile header lists what to remove once a chainsaw release
carries the fix. The README sentence added here has to go too, and so
does the Renovate rule that keeps the builder on Go 1.26.

The upstream PR also breaks one chainsaw test outside the packages it
touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks
the new `; operation terminated with error: context deadline exceeded`
suffix. I found it with a full `go test ./...` on v0.2.15 with and
without the patch. The image build tests only the patched packages, so
that test does not run here.

One trade-off: when the API stays broken, an assert or `error` step now
runs to its timeout instead of failing within seconds, so a red run
caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw
installed on its runner, not the sandbox image, so it does not get this
fix.

Known gaps, left for later: the new test covers only an assert on a
named object, not one that lists objects or an `error` operation, and
its skip guard does not notice if no test runs at all. The patch file
has no `-whitespace` entry in `.gitattributes`, so `git diff --check`
reports whitespace inside the upstream patch. A multi-platform build of
the image would run the builder tests once per platform; the image is
built for amd64 only.

Closes #4720

### Screenshots

Not a UI change.

### Downstream repositories

None of them builds or uses the e2e sandbox image; the change stays
inside this repo's CI.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* End-to-end sandbox assertions now retry after transient Kubernetes API
errors instead of failing immediately. Persistent errors still cause
assertions to fail when their timeout expires.
* **Testing**
* Added checks that verify assertions recover after temporary API errors
and fail within the expected timeout when errors persist.
* **Maintenance**
* Updated sandbox image build and dependency update settings for the
Chainsaw testing tool.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:25:39 +03:00
Aleksei Sviridkin a3470391e2 fix(linstor): mark volumes restored from a snapshot DRBD-initialized (#4866) 
## What this PR does

Restoring a PVC from a LINSTOR VolumeSnapshot can leave one replica with
an empty volume that DRBD marks UpToDate. A pod that lands on that
replica sees an empty filesystem.

linstor-csi restores the snapshot on one node and autoplaces the other
replicas. LINSTOR v1.33.3 copies `Linstor/Drbd/InitialUptoDateOn` from
the source volume into the restored volume definition and leaves the
definition without `DRBD_INITIALIZED`. When autoplace puts a replica on
the node that name points to, that empty new volume wins the initial
UpToDate race, gets a random current UUID and a fresh filesystem, and
goes StandAlone against the restored replica.

LINBIT fixed this in v1.34.0. This PR backports that commit,
LINBIT/linstor-server@9d8cdf5, to our v1.33.3 build as a patch. The
README entry says to drop the patch with the move to v1.34.0 or later.

Tested on a three-node cluster with a thick ZFS pool and
`placementCount: 3`. Before the change the reproduction failed every
time. With the patch I ran 5 restores, 4 of them with the source
volume's winner on a different node than the restore. I also restored a
snapshot taken before the change, and a 10Gi volume. Every time all
replicas came up connected and UpToDate, a pod on each node read the
same checksum, and the backing zvols were byte-identical when read with
`O_DIRECT`. The resync after a restore moves about the written data,
around 40 s for 1Gi and 134 s for 7.5 GiB on 10Gi. CSI clone PVC-to-PVC
was checked too and is not affected.

There is no e2e test. Every e2e lane runs without DRBD
(`COZY_LINSTOR_DRBD_ENABLED=false`, StorageClass `local`), so a suite
for replicated restores would never run in CI.

Volumes restored before this fix are not repaired by it. How to find and
fix them is in [this
comment](https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441).

Known limitation, shared by upstream v1.34 since that code did not
change: the autoplaced replicas skip the initial sync and rely on the
source volume's day0 UUID still being in the restored replica's history.
If that ever fails, DRBD goes StandAlone with "unrelated data" instead
of syncing, which is visible, not silent. I could not reproduce it.

Closes #4862

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): a PVC restored from a VolumeSnapshot no longer gets a replica with an empty volume marked UpToDate. Volumes restored before this release may still have such a replica; https://github.com/cozystack/cozystack/issues/4862#issuecomment-6106661441 explains how to check and repair them.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Restored snapshots with diskful DRBD resources are now recognized as
initialized by default, helping them resume without unnecessary metadata
recreation.
* When force-initial-sync is enabled, restored volumes continue through
the existing metadata-recreation path.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:25:22 +03:00
Aleksei Sviridkin d060c1dbb0 fix(kafka): pass the container limits to the Kafka JVMs (#4865) 
## What this PR does

The Kafka JVMs we ship size themselves from host memory instead of their
container limits, and get OOMKilled. The Strimzi 0.45.1-rc1 images carry
JDK 17.0.15, which reads `/proc/cgroups` to decide whether it runs in a
container. Kernels 6.12 and newer built without `CONFIG_MEMCG_V1` no
longer list the memory controller there, so the JVM turns container
support off. The JDK fix is JDK-8347811, backported to 17.0.20 and
21.0.10.

In CI this shows up in two places. The cluster operator gets a 25 GiB
max heap inside its 512Mi limit. In 107 of the last 150 e2e reports the
kernel OOM-killed a Java process at about 500M, the limit the cluster
operator shares with the topic and user operators. Brokers and
controllers start with a 5 GiB heap inside a 1Gi limit, because Strimzi
asks the same JDK for the container memory and takes half of what it
reports.

This PR passes the limits to every Kafka JVM explicitly:

- the cluster operator gets `-XX:MaxRAM` and `-XX:ActiveProcessorCount`
from its own limits through the downward API;
- brokers, controllers, the topic and user operators and the drain-time
Cruise Control get `JAVA_TOOL_OPTIONS` with `-XX:MaxRAM` set to their
memory limit, plus `-XX:ActiveProcessorCount` where the chart sets a CPU
limit. The chart sets one only for brokers and controllers. A LimitRange
default on the entity operators or Cruise Control is invisible to the
chart, and Strimzi template env takes no `resourceFieldRef`, so those
JVMs still count host CPUs.

Every java in a container reads `JAVA_TOOL_OPTIONS`, including the heap
probe Strimzi runs before starting Kafka. So Strimzi's own heap rules
and Kafka's G1 defaults apply to the real memory limit, and the chart
does no heap arithmetic. `jvmOptions` with `-XX` would not work here:
Strimzi turns those into `KAFKA_JVM_PERFORMANCE_OPTS`, and Kafka skips
its G1 settings when that is set. Each JVM now prints a "Picked up
JAVA_TOOL_OPTIONS" line on start.

In a kind cluster on such a kernel the operator then runs Serial GC with
a heap of about 100 MiB, and peaks at 249Mi while creating a Kafka,
against 440Mi before. A broker with a 1Gi limit and one CPU gets the
same `-Xmx518982205` that Strimzi computes when the JDK sees the limit,
with G1 and one concurrent GC thread.

Existing Kafka clusters roll once on upgrade, Talos ones included,
because the pod environment changes. Their heap does not change on a
kernel where the JDK already sees the limit.

The chart has no values key for JVM options or heap, so there is no user
setting this could override.

The Kafka migration hook now applies with `--openapi-patch=false`. Its
container runs under a tenant LimitRange default of 128Mi, and a plain
`kubectl apply` of an existing pool downloads the cluster's whole
OpenAPI document, about 300Mi with the platform's CRDs, so every upgrade
in such a tenant failed on an OOM-killed hook. With the flag kubectl
builds the same merge patch from the last-applied annotation and peaks
at about 35Mi. A larger memory limit would count against the tenant
quota, and a server-side apply would stop removing fields that `kubectl
create` set on the `kafka` pool. Other Jobs without resources are
tracked in #4868.

The operator OOMKill is what triggered the e2e failure in #4855. The
other half is a Strimzi race. After a restart, the PodSet controller
drops the ADD event of a PodSet that has no pods yet and only picks it
up at the 5-minute resync. That one is upstream and is described in the
issue. The other JVMs we ship are tracked in #4864.

Talos nodes are not affected, the Talos kernel keeps
`CONFIG_MEMCG_V1=y`. Generic-variant clusters on newer distribution
kernels are. CI is affected even though it runs Talos, because there the
Talos nodes are containers on the runner's Ubuntu 6.17 kernel.

Closes #4855

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kafka): the Strimzi cluster operator, Kafka brokers and controllers, the entity operators and the drain-time Cruise Control no longer size their JVM heap from host memory on kernels built without cgroup v1 memory support, which got them OOMKilled. The Kafka migration hook no longer runs out of memory on upgrade in a tenant with resource quotas. Existing Kafka clusters roll once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Kafka components now size JVM memory and processor settings based on
their configured resource limits, including during migrations.
- The Kafka cluster operator now receives container resource limits for
JVM sizing.

- **Bug Fixes**
- Kafka migration operations avoid OpenAPI patch lookups that could use
excessive memory.

- **Tests**
- Added coverage for JVM sizing, resource limits, and migration
behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:25:04 +03:00
Aleksei Sviridkin 1f74ad4ee7 fix(multus): drop the CPU limit and make resources configurable (#4830) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The multus daemon was too small to recover a node after a CNI outage,
and its limits could not be raised from values. The 100m CPU limit and
the 1Gi memory limit of `kube-multus` were hardcoded in the patched
upstream template.

On a node with about 600 dead sandboxes and 220 pods waiting, multus was
throttled in every CFS period. Its calls took longer than kubelet's
two-minute CRI deadline, so every call was retried and the queue only
grew. With the CPU limit lifted it OOM-looped at 1Gi. With 2 CPU and 4Gi
it cleared the backlog in about three minutes.

Now the resources come from a new `resources` value. The default keeps
the same requests (100m CPU, 100Mi memory), has no CPU limit, and raises
the memory limit to 4Gi. The memory limit stays on purpose. Without it
the pod cgroup has no `memory.max`, and multus becomes a candidate for
the Talos userspace OOM handler again.

The chart now refuses to render when an override removes the memory
limit or the memory request. Without a request Kubernetes uses the limit
as the request, so a node with less allocatable memory than the limit
would be left without multus.

The change is in `patches/customize-deployment.patch`, so `make update`
keeps it. I applied the patch to upstream v4.3.0 and got the committed
template, except the image references. The platform reverts a direct
edit of the child `cozystack.multus` Package, so a running cluster sets
the value through the new `networking.multus.resources` key of the
platform Package. The key is passed on only when set, and a value that
is not a map stops the render. The system-defaults LimitRange in the
namespace sets memory only, so it does not bring a CPU limit back.

Closes #4829

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/747
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds a platform value, so the hand-written key table on the
website needs it. The follow-up PR there is a draft until this one
merges.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(multus): the kube-multus container no longer has a CPU limit, and its memory limit is raised from 1Gi to 4Gi, so a node with hundreds of pods can recover after a CNI outage. Its resources can be overridden through the new `networking.multus.resources` platform value. An override that removes the memory limit or the memory request now fails the render.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional CNI plugin staging during Multus startup, configurable
through chart values. Staging skips when the plugin directory is
unavailable and reports individual copy failures while continuing.
* Made Multus resource settings configurable, with defaults of 100m CPU
and 100Mi memory requests and a 4Gi memory limit. Rendering requires
memory requests and limits.
* Updated the Multus deployment to use the `cozy-multus` namespace and
DaemonSet name, and Cilium as the master CNI configuration.
* **Bug Fixes**
  * Added a 128Mi memory limit to the CNI installation step.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:24:46 +03:00
Aleksei Sviridkin fa9f09e3d8 fix(vm-disk)!: refuse a golden-image clone CDI would leave unbound (#4796) 
## What this PR does

A VM disk created from a golden image into a StorageClass other than the
image's one can hang forever: its temporary clone PVC stays unbound, and
the user sees nothing. With the default `csi-clone` strategy, CDI checks
the source and target classes for a common provisioner, a matching
volume mode and room to expand. It never compares class names. So a
target class backed by the same provisioner gets no host-assisted
fallback, and the clone is left to the CSI driver. LINSTOR creates it
next to the source data, so on a target class that does not allow remote
volume access from every node (a node-local class, for example) the PVC
may never bind.

The vm-disk chart now refuses such a disk at render time and tells the
user which StorageClass to set. The check follows CDI's own clone
planner. The effective strategy comes from the CDI override, or else
from the target StorageProfile. It has to be `csi-clone`, or `snapshot`
when CDI gets a VolumeSnapshotClass of the provisioner. Once the
snapshot path has such a class it runs the same checks as csi-clone, and
LINSTOR ships one. Both classes have to share a driver (the golden's is
read from its PV, as CDI does), the volume modes have to match, and the
target class has to allow expansion. On top of that the chart refuses
only a LINSTOR target whose `allowRemoteVolumeAccess`, read the way
linstor-csi reads it, is not true; without the parameter that means a
layer list with no DRBD or NVMe layer. A remote-accessible LINSTOR class
and other drivers render. When the provisioner or the volume mode
differs, CDI falls back to a host-assisted copy, which completes, so the
disk renders as before. A target class without expansion is left alone
too, see the follow-ups.

The class check compares nothing when a lookup it needs returns nothing:
under `helm template`, before the golden PVC exists, or without the CDI
resource. An empty `storageClass` is compared as the class CDI would
pick for a kubevirt DataVolume, which is the default virt class,
otherwise the Kubernetes default class. Existing disks are exempt,
because their DataVolume spec is immutable. A disk counts as existing
when its DataVolume or its PVC exists, since CDI can garbage-collect a
completed DataVolume. Under `snapshot`, a snapshot class that both
StorageProfiles name but that does not exist is refused too, for any
driver, because CDI then fails to plan the clone instead of falling
back.

This is a breaking change: a new disk in a LINSTOR class without remote
access from every node rendered before and is refused now, and so is one
whose StorageProfiles name a snapshot class that does not exist. The
first case can include a clone that would happen to work: on a class
with `Immediate` binding, which I have not verified, or when the node
the clone is placed on already holds a replica of the golden image. The
way out is the golden image's class or the `copy` clone strategy.

How this covers the three fix directions in the issue:

1. Omitting `cloneStrategyOverride` when it is empty already landed on
main in #4138: the template skips the field for an empty or null value,
the values schema accepts the empty string, and the chart tests pin
both. This PR does not touch it.
2. I kept `csi-clone` as the default on purpose. Same-class clones are
the common case, and `csi-clone` is the fast path for them. Changing the
default is a separate decision, tracked in #4850.
3. vm-disk refuses the mismatch where the clone can hang, as described
above. That is narrower than the issue proposed, which was to refuse
every mismatch: every case CDI falls back on completes, and a clone into
a remote-accessible LINSTOR class should complete too. The last part
comes from reading the CDI and linstor-csi sources, I have not run it on
a cluster.

Follow-ups this PR does not cover:

- When several classes claim the default, CDI picks the newest one. The
chart does not reproduce that tie-break and compares nothing in that
case.
- The chart does not compare sizes. A target class without volume
expansion is never refused for remote access, even when the disk is no
larger than the golden image and CDI would attempt the clone.
- Under `snapshot` on a cluster that does not serve the snapshot API,
StorageProfiles naming a snapshot class still make CDI fail, and the
chart does not look that class up.
- A disk cloned from another disk (`source.disk`) into another class of
the same provisioner can hang the same way, and nothing refuses it yet.
- Whether `csi-clone` should remain the platform default now that it can
be turned off: #4850.

Closes #2908

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

No values, schema or ApplicationDefinition change. Re-check notes for
the next CDI and linstor-csi bumps sit next to the version pins in the
kubevirt-cdi-operator and linstor Makefiles. The new note in the vm-disk
README reaches the website through the regular reference-page
regeneration.

### Release note

```release-note
fix(vm-disk)!: a new disk from a golden image is now refused with a clear error when CDI would CSI-clone it into another LINSTOR StorageClass that does not allow remote volume access from every node. Before, such a disk could stay pending forever with no error. A clone that would have landed on a node already holding a replica of the golden image is refused as well; use the golden image's StorageClass or the copy clone strategy.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Prevented image-backed disks from being rendered when CDI’s selected
clone strategy cannot safely clone across differing storage classes,
including applicable LINSTOR remote-access and snapshot-class checks.
* Existing DataVolumes and same-namespace PVCs are not blocked by these
checks. Disk creation can proceed when required information is
unavailable or the selected strategy supports the operation.
* **Documentation**
* Clarified when image-backed disks may be refused and recommended using
the golden image’s storage class or the copy strategy as alternatives.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:24:29 +03:00
Aleksei Sviridkin 938abdc4d2 fix(platform)!: refuse a networking variant change on a live cluster (#4790) 
## What this PR does

Changing the platform variant on a running cluster can delete its CNI,
and nothing stopped that before. A new platform variant (`spec.variant`
of the `cozystack.cozystack-platform` Package) can give the
`cozystack.networking` Package a different variant, and the operator
then deletes every HelmRelease of that Package the new variant does not
install. Moving from `isp-full` to `isp-hosted` deletes Cilium and
Kube-OVN, so every pod loses networking. Moving from `isp-full` to
`isp-slim` deletes Kube-OVN.

With this PR the platform render looks up the live
`cozystack.networking` Package. If its variant differs from the one
about to be rendered, the render fails, and the error names the value
that lets it through: `networking.acknowledgeVariantChange: "<new
variant>"`. I made the acknowledgement a variant name instead of a
boolean like `publishing.proxyProtocolAcknowledgeUnclean`. A leftover
`true` would let every later switch through unchecked, while a variant
name covers only the switch it names.

The guard does nothing when there is no live Package to compare against:
`helm template`, a client-side dry-run, or a first install. It is also
skipped when `cozystack.networking` is listed in
`bundles.disabledPackages`, because the platform does not render the
Package then. It lives in the shared Package helper and not in
`bundles/system.yaml`, so it covers both places that emit the networking
Package, including the `noop` call in the `isp-hosted` branch. The other
Packages whose variant follows the bundle, `cozystack-scheduler` and
`tenant-application`, keep the same release names in every variant, so
changing their variant deletes nothing and they need no guard.

The new helm-unittest cases cover the refused switches, an unchanged
variant, a matching and a mismatching acknowledgement, a bare boolean,
`disabledPackages`, an offline render, and a live cluster with no
networking Package yet.

A cluster where someone set the Helm-owned networking Package to
`cilium-kilo` by hand now fails its platform upgrade, and the error
points it at `bundles.disabledPackages`. Acknowledging does not help
there, because the render writes the bundle's variant over
`cilium-kilo`. helm-controller v1.5.0 installs a release with
server-side apply by default, which takes over every rendered field. On
a release it installed, the upgrade replaces the variant and the
operator deletes Kilo. A release first installed with Helm 3 keeps the
apply method recorded in its history, so there the outcome depends on
that history.

Known gap, left for a follow-up: the guard fires on any variant change,
including Talos to generic (`kubeovn-cilium` to
`kubeovn-cilium-generic`). Those pairs share release names, so nothing
is deleted and only values change. Refusing is safe, and the error does
not claim a deletion for such a switch.

Closes #4596

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

This PR adds a key to `packages/core/platform/values.yaml`, and the
website keeps a hand-written table of those keys in
`operations/configuration/platform-package.md`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/746
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): the platform render now refuses to change the variant of a live cozystack.networking Package, because the operator would then delete the CNI HelmReleases the new variant does not install (isp-full to isp-hosted removes Cilium and Kube-OVN). To go ahead with such a change, set networking.acknowledgeVariantChange to the new networking variant named in the error. A cluster whose cozystack.networking variant was set by hand (for example cilium-kilo) needs cozystack.networking in bundles.disabledPackages before the upgrade, otherwise the platform render refuses.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Changing the networking variant on a running cluster now requires
explicit confirmation naming the requested variant. A change without a
matching confirmation is blocked.
* The confirmation applies only to the specified switch and must be
updated for a later variant change. If the running package has no
recorded variant, it is treated as using the default variant.
* New installations without an existing networking package are
unaffected, and networking can still be disabled through its package
setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:24:09 +03:00
Aleksei Sviridkin 953b965e66 fix(kubernetes): keep NFS publish from hanging on a dead server (#4791) 
## What this PR does

When the kubevirt-csi node plugin publishes an NFS volume, it first
mounts the export root into an `nfs-init-*` temp dir to create `/data`
and move old files into it. That temp mount was hard. If the NFS server
stopped answering at that point, the publish call never returned.
kubelet gives up after 2m and retries, each retry blocked the same way,
and goroutines and temp dirs piled up until the server came back or the
plugin restarted.

The fix does three things:

1. The temp root is now mounted `soft` with `retry=0`. It only does
metadata work that a retried publish redoes, so an error there is
harmless, and a stuck call now ends on its own once the connection to
the server is declared dead. The mount the pod gets stays hard, because
a soft mount can report a failed write that the server did apply.
2. A publish to a target path that already has one running now returns
`Aborted` instead of starting another. This is what stops the pile-up:
at most one call per target path is stuck at a time.
3. Moving old files into `/data` now checks the destination with `lstat`
and fails the publish on any error other than a missing file. With the
soft mount, the check can fail with EIO on a file that `/data` already
holds, and a rename issued after the server comes back would replace
that file. `lstat` keeps a user's symlink in `/data` that loops or
dangles from failing every publish. Such a symlink counts as present and
is left alone, while before this change the rename replaced a dangling
one.

I kept the default retry timeouts on purpose, although the issue
suggested lowering `timeo` and `retrans`. The Linux client shares one
connection per server for all of that server's mounts on the node, and
the first mount sets the connection's TCP timeouts. Lowering them on the
temp mount would have changed the timeouts of the pods' hard mounts too.
Over NFSv4, a soft RPC fails only after the connection is declared dead,
so a stuck call takes minutes to end, longer than kubelet's 2m deadline.
The guard keeps kubelet's retries from adding new calls during that
time.

The guard works per target path, not per volume. Two pods on one node
that use the same RWX volume still publish in parallel, and each pod can
have at most one stuck call.

The unit tests pin the mount options on both mounts, the guard behaviour
including its release after a failed publish, and the destination check:
a failed check stops the move, and a symlink in `/data` stays in place.
They use the fake mounter. I did not test against a real NFS server that
went away.

Not covered here, and tracked in #4814:

- If the server's TCP connection stays up while the NFS service hangs, a
soft NFSv4 RPC does not time out. The guard still limits it to one stuck
call per target.
- `Mount` from mount-utils runs `mount` with no deadline. This predates
this change, and the guard bounds it the same way.
- If the target is already hard-mounted from the dead server, the check
of the target at the start of a publish blocks for good. The guard keeps
this to one call per target but does not end it. Unpublish is not
coordinated with a publish that is still running either. Both predate
this change.
- A follow-up comment on the issue reported two leftovers of the bounded
unmount: an orphaned `umount.nfs` after a timeout, and a possible `stat`
before `umount2`. Both may end sooner now that the temp mount is soft,
but I have not verified either.

Refs #4581

### Screenshots

Not a UI change.

### Downstream repositories

The change is limited to the kubevirt-csi node plugin's Go code. None of
the triggers in the contributing guide's map apply: no package added or
renamed, no platform values, variants, release assets,
ApplicationDefinition semantics or developer tooling touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): an NFS volume publish in tenant Kubernetes clusters no longer hangs forever when the NFS server goes away during the publish, and kubelet retries no longer pile up while a publish is stuck. The plugin's temporary mount of the export root is now soft, and a second publish to the same target path is refused while the first is still running.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Concurrent NFS publish requests for the same target now return an
`Aborted` error. Different targets remain available, and a target can be
used again after a request finishes, including after a failed attempt.
* Migration now stops if checking a destination encounters an unexpected
error. Existing destinations, including symlinks, are left in place.
* Temporary-directory creation and mount errors now identify the
affected volume.
* **Updates**
* Temporary NFS root mounts now use soft behavior with retries disabled.
The final data mount remains hard.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:23:47 +03:00
Aleksei Sviridkin b86513800f docs(governance): welcome new contributors and link LF sanctions guidance (#4845) 
## What

- Adds a "Welcoming new contributors" section to `GOVERNANCE.md`:
response expectations, help with DCO, a safety read of the diff before
every CI approval for external contributors, and how to review and
follow up.
- Adds one line to the maintainer expectations pointing to the Linux
Foundation guidance on OFAC sanctions and open source, with questions
raised privately among maintainers.

## Why

Maintainers had no written guidance for first contributions. For
sanctions, the LF guidance applies to all LF projects, so the document
links to it instead of restating it.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Expanded maintainer guidance for welcoming new contributors, including
responding within two working days, helping with contribution and DCO
requirements, and suggesting a next step.
* Clarified expectations for consistent reviews, explaining decisions,
and checking external contributors’ CI changes for workflow, secret, or
release-tool risks before approval.
* Added guidance on Linux Foundation advice concerning OFAC sanctions
and raising related questions privately with other maintainers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 17:10:57 +03:00
tym83 b524a42ae7 docs(governance): welcome new contributors and link LF sanctions guidance 
Maintainers had no written guidance for a first-time contributor. Add a
short section: reply within two working days, help with the DCO, read
the diff before approving any CI run for an external contributor, review
to the same standard as any other change, and say early when a change
will not be accepted.

Sanctions questions are left to the Linux Foundation guidance, which the
maintainer expectations now link instead of restating it.

Assisted-by: LLM
Signed-off-by: tym83 <6355522@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-12 16:08:37 +03:00
Aleksei Sviridkin 785cbc391b fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (#4293) 
## What this PR does

Fixes #4060.

`flux-plunger` recovers a HelmRelease stuck with `has no deployed
releases` by briefly suspending it, deleting the stale Helm storage
Secret, recording a processed-version annotation, and unsuspending.
helm-controller v1.5.x skips the Helm uninstall and drops its finalizer
when it handles a delete on a suspended HelmRelease, and it wakes on the
same delete event flux-plunger does, so a delete arriving inside that
window orphaned every resource the release owns (for a VMDisk:
DataVolume, PVCs, the CDI importer pod, RBAC, WorkloadMonitor and the
storage Secret itself). Lifting the suspension on the delete event
cannot win that race reliably. The bug is not specific to VMDisk.

The change makes such a delete reach the uninstall:

- **An uninstall fence.** The suspension is applied together with a
flux-plunger finalizer, `flux-plunger.cozystack.io/uninstall-fence`, in
one server-side apply, so flux-plunger never holds a suspension without
it. If a delete arrives while the release is suspended, the fence keeps
the object whether helm-controller or flux-plunger handles the delete
first: flux-plunger lifts its suspension and keeps the fence,
helm-controller reconciles the unsuspended generation and runs the
uninstall, and flux-plunger removes the fence once helm-controller's
finalizer is gone and it has observed that generation. A recovery that
completes lifts the suspension and the fence together. A terminating
HelmRelease never begins recovery, and termination is re-checked
immediately before deleting the storage Secret.
- **Only flux-plunger's own suspension is acted on.** Ownership is
decided by which field manager owns `spec.suspend` in
`metadata.managedFields`: the apiserver transfers the field away the
moment an operator (or `cozyhr resume`, or a backup/restore controller)
writes a different value to it, so a later suspension is never misread
as flux-plunger's. A suspension set through the shared
`flux-client-side-apply` manager (platform migrations, the tenant delete
hook, an operator's manual `kubectl apply`) is never claimed. A release
deleted under another manager's suspension is that manager's decision:
flux-plunger does not hold it with the fence, and helm-controller leaves
its resources in place, as the migrations that suspend and then delete
rely on.
- **No suspension taken over in a race.** Acquiring applies without
forcing first and, on a conflict, re-fetches and forces only if
`spec.suspend` is still `false`: a backup controller owns the field at
`false` via a dynamic-client `Update`, which a non-forced apply can
never take. Every apply flux-plunger sends, acquiring as well as lifting
the suspension or the fence, is preconditioned on the `resourceVersion`
and `uid` of the read that decided on it. The `resourceVersion` makes a
write landing in between fail the apply instead of being absorbed:
acquire then defers rather than adopt or take over a suspension set
meanwhile, and lifting the suspension and the fence from a release read
as live cannot delete it without an uninstall if helm-controller has
handled a delete in the meantime. The `uid` stops an apply to a release
that has gone from creating an empty one, since server-side apply
creates what does not exist. A conflict is retried through the requeue.
The apply carries only `spec.suspend` and the fence, so it cannot leak
the required `spec.interval` and conflict on it.
- **Fresh reads.** The release, every re-check and the storage Secret
list are read from the apiserver (an uncached `APIReader`), not the
informer cache. A lagging cache would let the processed-version guard
delete the revision below the one just reclaimed, and the ClusterRole
grants secrets `get`/`list` but not `watch`, which an informer needs to
stay synced; flux-plunger deliberately does not cache Secrets. Every
step requeues with backoff on failure.
- **Events.** A Warning when flux-plunger holds a suspension it failed
to clear, and a Warning when a delete has been held by the fence for
more than ten minutes waiting for helm-controller (the held state is
re-checked on a timer, not only on helm-controller's next status write).
A release deleted under another manager's suspension gets a Normal event
when that is a migration's choice, and a Warning when the release
carries flux-plunger's processed-version annotation, because then the
skipped uninstall is an accident, not a decision. Normal events also
mark an externally-owned suspension left alone and a recovery deferred
because the release changed between read and write; an external
suspension on a release flux-plunger has recovered before is a Warning
instead, because it is most likely one an earlier flux-plunger left
mid-recovery (see Operating notes). Events from the shutdown drain are
written straight to the apiserver, since the manager's recorder stops
with the manager.
- **The fence is released when flux-plunger goes.** Once its manager has
stopped, and with it every reconcile, flux-plunger drains its fences
under a deadline inside the pod's termination grace period; a stop that
overran its 10s bound is drained too and still reported: a recovery in
progress is aborted, the suspension and the fence lifted together for
the next flux-plunger to start again, and a release being deleted has
flux-plunger's suspension lifted and keeps the fence until
helm-controller has run its uninstall. A release helm-controller does
not handle in time keeps its fence. The drain lives in the controller,
so the envtest suites hold it against a real apiserver's generations and
finalizers. A pre-delete hook on the chart is the backstop for what the
drain could not finish, so uninstalling the component never leaves a
release it cannot finish deleting: it stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and fence without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, the outcome the controller's own relinquish has, with a JSON
patch of `spec.suspend` preconditioned on the `resourceVersion` and
`uid` it read; not with `kubectl apply --server-side`, which on a
release last written by a client-side `kubectl apply` moves that apply's
fields to the applying manager and strips every one the body leaves out.
helm-controller runs a pre-delete hook under the release's uninstall
timeout, 300s by default and not set by the operator, so the Job's
deadline sits under it. A Deployment that cannot be stopped or a listing
that never succeeds ends the hook before any fence is touched, and a
fence it cannot remove ends it after the sweep; either way it exits
non-zero, because a failed pre-delete hook leaves the release untouched
and helm-controller retries the uninstall, hook included, whereas
exiting 0 would let the component go and strand the fence. A
flux-plunger pod still there after its grace period, typically one stuck
`Terminating` on a `NotReady` node, is only reported, and the pod wait
is cut to what is left of the hook's retry window; because such a pod
may still be setting fences, the fences are listed again after the sweep
and any found fails the hook. The Job runs once and is kept after
success so its report stays readable until its TTL;
`helm.sh/hook-output-log-policy` is deliberately not used, because
helm-controller never sets Helm's hook output writer and calls a nil one
for it (fluxcd/helm-controller#1562). The hook's kubectl image is
digest-pinned in the chart values, and its RBAC is only what `kubectl
scale`, the listing and the patches issue. The chart's README documents
the finalizer and how to clear it by hand.

A single writer is kept by leader election, not by the rollout. A second
process acting under the same field manager could clear a suspension the
first is still recovering, and no Deployment strategy prevents one: the
next pod starts as soon as the old one is terminating (Kubernetes v1.35
has no pod-replacement policy for Deployments), and an eviction or a
node drain replaces a pod the same way, while the old one stops and
drains. The chart now passes `--leader-elect` with a Role on `leases` in
the release namespace. The Lease is not given up on stop and outlasts
the last renewal, the renew deadline, the manager's bounded stop (10s)
and the drain, and only a flux-plunger that led drains at all, so the
next one acts only once the old one is done; the price is that a new pod
waits for the Lease to run out, a little under two minutes at the
default. The timings derive from one value, `shutdownGracePeriodSeconds`
(90 by default, refused outside 40 to 140 at render): the pod's
termination grace period, the drain deadline (25s less: the manager's
10s stop plus a 15s margin before the kubelet's SIGKILL, both tied to
the Go constants by a contract test) and the hook's pod wait. The
strategy keeps `maxSurge: 0` and `maxUnavailable: 1` and stays
`RollingUpdate` rather than `Recreate` on purpose: the chart shipped
without a strategy block, so every existing install's live Deployment
carries the apiserver-defaulted `rollingUpdate` object and
helm-controller applies server-side, where laying `type: Recreate` over
that populated block is rejected on upgrade.

### Operating notes

A HelmRelease carries the fence only while flux-plunger is recovering it
or waiting for helm-controller to uninstall it. A fenced release whose
uninstall keeps failing stays `Terminating` exactly as any HelmRelease
does under helm-controller's own finalizer, and raises the Warning
above. Stopping flux-plunger drains its fences, and uninstalling it
releases whatever the drain left through the pre-delete hook; a fenced
release that is deleted while flux-plunger is merely scaled to zero, or
killed without a graceful stop, and that helm-controller has not
uninstalled, stays `Terminating` until flux-plunger runs again. To
release one by hand, after confirming its uninstall has run or is not
wanted:

```bash
kubectl patch helmrelease NAME --namespace NAMESPACE --type json --patch '[{"op":"test","path":"/metadata/finalizers/INDEX","value":"flux-plunger.cozystack.io/uninstall-fence"},{"op":"remove","path":"/metadata/finalizers/INDEX"}]'
```

where `INDEX` is the position of
`flux-plunger.cozystack.io/uninstall-fence` in `metadata.finalizers`;
the `test` operation makes the patch fail rather than remove another
finalizer if the list has changed.

Earlier flux-plunger releases set `spec.suspend` through the shared
`flux-client-side-apply` manager, so a suspension one of them left
behind cannot be told apart from a migration's or an operator's and is
treated as external: it is not cleared automatically. Such a leftover
exists only if the old pod stopped in the middle of a recovery, and
flux-plunger raises a `SuspensionNotOwned` Warning on it. The chart
README carries this section too. To list candidates, suspended releases
carrying the flux-plunger annotation whose `spec.suspend` is not owned
by flux-plunger's apply:

```bash
kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | jq --raw-output '.items[] | select(.spec.suspend == true) | select(.metadata.annotations["flux-plunger.cozystack.io/last-processed-version"] != null) | select([.metadata.managedFields[]? | select(.manager == "flux-plunger" and .operation == "Apply" and .fieldsV1["f:spec"]["f:suspend"] != null)] | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"'
```

Clearing one is an operator action: confirm nothing else (a migration, a
restore) is holding it, then resume it with `flux resume helmrelease
NAME --namespace NAMESPACE`.

### Testing

The regression the issue asks for is
`TestEnvtest_DeleteDuringRecoveryReachesUninstall`: against a real
apiserver, a delete lands while flux-plunger holds its recovery
suspension, and helm-controller's v1.5.0 delete handling is reproduced
step for step (uninstall only when not suspended, finalizer dropped
either way, observed generation recorded). It covers both orders,
helm-controller handling the delete first and flux-plunger handling it
first; without the fence the first order leaves the release gone with no
uninstall. The same suite pins the managedFields ownership dynamics a
fake client cannot represent: the field transfer, the forced acquire
over a dynamic-`Update` owner, the `resourceVersion` precondition
rejecting a write that races either apply, the boundary that a
`flux-client-side-apply` suspension is never claimed, and interval
preservation. The fence mechanics were also checked against a live
helm-controller: with another finalizer holding a deleted, suspended
release, lifting the suspension made helm-controller run the uninstall
and remove the release's resources.

The envtest suite runs on every pull request: a new `make
test-controllers-envtest` target builds a pinned `setup-envtest` and
fetches a kube-apiserver and etcd into `_out/envtest`, which CI caches
and the target reuses without asking the release index, takes the
HelmRelease CRD from the in-tree Flux manifests, and CI calls it next to
`make test-controllers`; the control plane gets a longer start timeout
because it starts beside the other test goals. The drain is held there
too: it aborts a live recovery, carries a delete through the uninstall
and lifts the fence only once helm-controller has observed the
unsuspended generation, keeps the fence when helm-controller never
answers, releases a fence left without a suspension at once, and leaves
another manager's suspension alone.
`TestEnvtest_TheNextFluxPlungerLeadsOnlyAfterTheDrain` runs two real
managers on one Lease: a clean stop still drains, and the second manager
becomes leader only after the first one's drain has ended; it goes red
if the Lease is given up on stop. Unit tests cover the controller logic
(every branch, ordering, requeue, and the ownership decision through an
injected seam), the Lease timing relations, that only a leader drains
and through the manager's API reader rather than its stopped cache, that
a failed listing is retried, that a stop which overran its grace period
is still drained and its events reach the apiserver, and contract tests
tie `main.go`, the chart's flags, the drain's arithmetic and the hook's
names to the Go code. The pre-delete hook is covered by helm-unittest
and by `hack/flux-plunger-fence-hook_test.bats`, which runs its rendered
script against a fake `kubectl` that keeps each release's finalizers,
suspension owner, generation and observed generation, bumps the
generation on a spec write as the apiserver does, answers NotFound
without `--ignore-not-found`, spends simulated time on waits, lists a
release once however many copies of the fence it carries, refuses
`kubectl apply`, lifts a suspension only through the preconditioned JSON
patch and never one another manager also holds, reads a negative
`kubectl wait` timeout as a week as kubectl does, and applies a
finalizer removal only when it is the guarded test-then-remove at an
index that holds the fence. Every mutation in a set covering each
predicate conjunct, each retry, each exit path, the re-listing, the
timings and the selector turns the hook suite red, as does every
mutation in the Go set covering the drain's branches, the Lease timings
and the events; three drain mutations that still survive are tracked in
#4782. The hook script was also run as a Job in the hook's own image on
a live cluster, succeeding, with the Deployment already gone, and with
patch denied by RBAC, where it exited non-zero and named the release it
left; that run caught that a per-call `--request-timeout` makes kubectl
drop its in-cluster config, so the hook sets none and the fake rejects
it. A later live run caught `kubectl apply --server-side` stripping the
finalizers and spec of a release created by a client-side apply; with
the JSON patch the same releases kept every field, the suspension
flux-plunger alone held was lifted, a co-owned and a foreign one stayed,
and every fence went.

### Remaining limitations

- A delete that arrives after recovery has deleted the storage Secret of
a release whose only revision was that one finds no revision to
uninstall; helm-controller then treats the release as already gone.
- A writer that sets `spec.suspend: true` while flux-plunger already
holds it at `true` does not take the field over, because the value does
not change. If such a writer then deletes the release inside the
recovery window (the suspend-then-delete that the tenant delete hook and
some platform migrations use to skip the uninstall), flux-plunger still
sees its own suspension, lifts it, and helm-controller runs the
uninstall that writer meant to skip. The same held before this change,
when the end of recovery unsuspended unconditionally.
- The already-processed guard compares revision numbers only. When Helm
re-creates a revision under the same number after recovery deleted it (a
release whose only revision fails again the same way), recovery runs
again for it.
- `listHelmReleaseSecrets` reads the local cluster only, so a
HelmRelease that targets a remote cluster via `spec.kubeConfig` (whose
storage Secrets live in the remote cluster) is not recovered; it is
skipped, never mis-handled.
- A new flux-plunger pod waits for the old one's Lease to run out before
it acts, on every restart and after a crash: a little under two minutes
at the default. Recovery is not latency-sensitive, and this is what
keeps a single writer while the old pod drains.
- The leader Lease is left in the release namespace after the chart is
uninstalled, as with any controller-runtime leader election; it is
inert.

### Screenshots

Not applicable: no UI change.

### Downstream repositories

The trigger map was walked against the diff: it touches
`internal/controller/fluxplunger`, `cmd/flux-plunger`, the
`packages/system/flux-plunger` chart (templates including RBAC for the
leader Lease, a pre-delete hook, values, tests, its package `Makefile`,
README), a new bats test under `hack/`, the root `Makefile` (a new
`test-controllers-envtest` target; existing targets are unchanged) and
the pull-request workflow that calls it and caches its binaries. No row
matches: ccp gates on existing `hack/` files and make targets, and
nothing there is moved, renamed or changed in behaviour; a new test file
and a new target reach none of them. The website's FluxPlunger section
describes the recovery steps, which still hold; it does not describe the
fence or the drain, which the chart README now documents.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(fluxcd): a HelmRelease deleted while flux-plunger is recovering it now reaches the Helm uninstall instead of leaving its resources (DataVolume, PVCs, RBAC, storage Secret) behind
```
 2026-10-12 15:23:49 +03:00
Mattia Eleuteri 0c62bee4da feat(bucket): add a ui flag to skip the s3manager web UI 
Every bucket ships an s3manager Deployment with its Service and an
Ingress or HTTPRoute on <bucket>.<tenant host>. Tenants that only use
the S3 API pay for a pod per bucket and for one DNS name and TLS
certificate per bucket, none of which they need.

ui defaults to true, so existing buckets keep their UI. With ui set to
false the bucket no longer needs the tenant apex either, since the UI
hostname was its only consumer in the bucket-system chart.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
 2026-10-12 13:09:23 +02:00
Aleksei Sviridkin 7c46ffff12 fix(backupstrategy-controller): run backup clients on a package-owned image (#4756) 
## What this PR does

The Altinity (ClickHouse) and RabbitMQ backup strategy Pods ran
`platform-migrations`: a 349 MB image carrying `kubectl`, `helm`, `git`,
`cozyhr` and `etcd-migrate`, whose `ENTRYPOINT` is `run-migrations.sh`.
Both pins were stamped from `packages/core/platform/Makefile`, so any
migration change re-digested the backup clients too, and only the
`command:` override kept the Pods from running platform migrations
(#3420).

- New
`packages/system/backupstrategy-controller/images/backup-http-client/`:
`alpine:3.24` (digest-pinned, the same base the tree already uses) +
`curl` + `jq` + `ca-certificates`, `USER 65532:65532`, no `ENTRYPOINT`.
Everything is installed at build time, so the Pods still never run `apk
add` and keep working air-gapped.
- New `image-backup-http-client` target in the package's own `Makefile`,
chained after `image-redis-backup` like the existing ones so only one
`yq -i` writes `values.yaml` at a time. It stamps `chBackupClientImage`
and `rabbitmqBackupClientImage` with the same ref.
- Removed the cross-package stamps from
`packages/core/platform/Makefile`, the WORKAROUND/TODO comments in the
chart and the RabbitMQ example, and the stale "live violation" line in
`docs/agents/image-refs.md`.
- `hack/image-pin-consistency.bats`: dropped the
`platform-migrations`-specific test, whose subject no longer exists. The
general check (no repository pinned at more than one digest, computed
from `hack/promote-retag.sh --dry-run`) still covers the two keys
sharing one image.

**Why one image for both strategies rather than `ch-backup-client` as
the issue named it.** The RabbitMQ strategy had the same reuse and the
same cross-package stamp; leaving it would have kept the stamp alive.
Its script needs `curl --aws-sigv4` + `ca-certificates`, a subset of the
ClickHouse one, so one image serves both. The two values keys stay
separate so either strategy can move to its own image later.

**Initial pin.** The image does not exist in ghcr until CI builds it, so
both keys are committed as `backup-http-client:v1.6.0` without a digest,
the same way `redis-backup` was introduced. PR CI rebuilds touched
packages and overlays the fresh refs for e2e; release preparation's
`make image` stamps tag+digest.

Image size: 6.9 MB vs 349 MB. Verified locally:
- `docker buildx build` for `linux/amd64,linux/arm64`; in the image as
uid 65532: curl 8.22 with `--aws-sigv4`, jq 1.8.2, `date`, `seq`, writes
to `/tmp` (the RabbitMQ script's only write), TLS verification against
the bundled CA store.
- `helm unittest` (23/23), `helm template` renders both strategies with
`backup-http-client` and no `platform-migrations` reference.
- `hack/cozytest.sh hack/image-pin-consistency.bats` and
`hack/build-matrix_test.bats` pass.

Not yet run end to end on a cluster; the e2e backup round-trips will
exercise it.

Fixes #3420

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff: it touches
`packages/system/backupstrategy-controller` (new image, Makefile target,
values comments/pins, template comments),
`packages/core/platform/Makefile` (stamp removal only, no `values.yaml`
key change), `hack/image-pin-consistency.bats`,
`docs/agents/image-refs.md` and a comment in
`examples/backups/rabbitmq`. No app package, schema, CRD, variant,
asset, platform values key or `hack/package.mk` change. The website does
not list first-party image repositories (searched for
`platform-migrations` and `redis-backup`), so no mirror list needs the
new one.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backupstrategy-controller): the ClickHouse (Altinity) and RabbitMQ backup strategy Pods now run a dedicated `backup-http-client` image (alpine + curl + jq + ca-certificates, non-root, ~7 MB) instead of the 349 MB `platform-migrations` image. Air-gapped mirrors need the new `ghcr.io/cozystack/cozystack/backup-http-client` repository.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Altinity and RabbitMQ backups now use a dedicated backup HTTP client
image, configured separately from the platform migrations image.
* The client image includes the tools required for backup operations and
runs with a non-root identity.
* Both backup configurations receive the same digest-pinned image
reference when the image is built.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 12:38:55 +03:00
Aleksei Sviridkin 8deaa8e14c fix(vm-disk): name the importer error in the disk's WorkloadsReady message (#4815) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A VMDisk whose HTTP source answers 404 or does not resolve never
finishes importing, and since #4380 its `WorkloadsReady` condition says
only `DataVolume vm-disk-x is ImportInProgress`. That reads the same as
a large download that is still running. CDI fails an import only on a
fatal error (a checksum mismatch) and restarts the importer pod
otherwise, so the phase never moves, and the reason the disk will not
finish was visible only in the importer logs.

CDI already keeps that reason on the DataVolume: the `Running` condition
goes `False` with the importer's termination message, and keeps it
through the back-off between restarts. The WorkloadMonitor now appends
it, so the condition reads `DataVolume vm-disk-x is ImportInProgress:
Unable to connect to http data source: expected status code 200, got
404. Status: 404 Not Found`. The console already shows the
`WorkloadsReady` message on the disk's overview page, so no UI change is
needed. The same text covers an unschedulable importer, a failed image
pull and a checksum failure, since CDI reports all of them through
`Running=False`.

Only `Running=False` is read: `Unknown` is not a failure, and `Bound`
repeats the same error behind PVC details. For a host-assisted clone CDI
joins the target and source pod messages as `<target> and <source>`,
which comes out as ` and ` when both pods are still being created, so
empty halves are dropped.

Behaviour to be aware of: while the importer pod is briefly running
between restarts, CDI flips `Running` to `True` and clears the message,
so the condition shows the bare phase for those seconds. This is not a
terminal state either: a disk with a broken source still stays
`ImportInProgress` and keeps retrying. Deciding when such an import
should count as failed is left for a separate change.

### Tests

A table test covers a 404, an unresolvable host, an unschedulable
importer, a fatal checksum error, a DataVolume without a phase and
joined clone messages with one or both halves empty, plus the cases that
must not add text: an importer that has not started, a running importer,
an `Unknown` condition, a populated disk and an error that only sits on
`Bound`. Each conjunct of the condition check and the append itself were
mutated and each mutant fails the test. A second test runs against a
DataVolume captured from a test cluster after its importer had restarted
about two thousand times on a 404, with only names and the URL changed,
so the parsing is pinned against what CDI actually writes rather than a
hand-built shape.

### Screenshots

No UI code changes: the console renders the `WorkloadsReady` message it
already receives. A message that carries a long source URL is not
wrapped on the overview card; that is a styling follow-up for the
console and is left out of this change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only non-Go change is a sentence in the
`packages/apps/vm-disk/README.md` notes; no values, schema, API kind or
default changes. The website reference page for vm-disk is regenerated
from that README by the release bot.

### Release note

```release-note
fix(vm-disk): the WorkloadsReady condition of a VM disk now includes the importer error reported by CDI, such as a 404 or an unresolvable host, so a disk whose source keeps failing shows why it is stuck instead of only ImportInProgress.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Workload status messages now include available DataVolume import
failure details, such as HTTP errors or DNS resolution problems, making
the cause of an import issue clearer.
* **Documentation**
* Updated the workload readiness documentation to explain when
DataVolume phase and import error details appear in status messages.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 12:29:50 +03:00
Aleksei Sviridkin 38d6ac7efb feat(redis): expose min-replicas-to-write and min-replicas-max-lag (#4848) 
## What this PR does

Adds two optional values to the Redis app, `minReplicasToWrite` and
`minReplicasMaxLag`, rendered as `min-replicas-to-write` and
`min-replicas-max-lag` in the RedisFailover `spec.redis.customConfig`.
Both are unset by default, so the rendered config and behaviour are
unchanged for existing instances.
Replication is asynchronous, so a master that is isolated from its
replicas keeps accepting writes until Sentinel fails over, and those
writes are lost. With `minReplicasToWrite: 1` the isolated master stops
accepting them after `minReplicasMaxLag` seconds, which bounds the loss.
Trade-off: with the default `replicas: 2` there is a single replica, so
the master also refuses writes whenever that replica is down (node
drain, restart, upgrade). Users who turn this on accept lower write
availability for a bounded loss window.
Includes helm-unittest cases and the regenerated schema, README, Go
types and ApplicationDefinition.


### Screenshots


### Downstream repositories


- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/68
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note


```release-note
feat(redis): add optional minReplicasToWrite and minReplicasMaxLag values to bound writes lost on failover; unset by default
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional Redis settings to require a minimum number of replicas
to acknowledge writes and to set the maximum age of those
acknowledgements.
* Invalid settings are rejected: the required replica count must be
nonnegative and below the configured replica count, and the
acknowledgement age must be at least 1 when a replica requirement is
set. Replication remains asynchronous, so these settings do not
eliminate potential write loss during failover.
* **Documentation**
* Documented the settings, defaults, validation, and behavior in Redis
configuration guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.7.0-rc.1 		2026-10-12 10:53:58 +03:00
myasnikovdaniil ebceef4105 ci(retention): sweep packages named by the published nightly tree (#4826) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The Artifact Retention workflow has failed every day since mid-August,
and it never pruned anything. It lists the org's container packages with
`GET /orgs/cozystack/packages?package_type=container`, and that endpoint
answers `400 Invalid argument` to the app installation token the job
uses. Others hit the same in actions/github-script#449 and cli/cli#9606.
An old `|| true` used to hide it. This is item A19 in #3278.

The per-package endpoints accept the same token: a probe run confirmed
it, and the dry-run below read every package with it. So the job no
longer lists the org. The new `hack/retention-packages.sh` takes the
package names from the published `cozystack-packages:nightly` tree,
reading refs through the shared `hack/lib/image-refs.sh`. It also adds
`cozy-installer` and `cozystack-nocloud`, which the nightly publishes
outside the tree. The committed tree can't be the source: it misses
images whose digest only the build pins, and those still get dated
nightly tags every day.

Failures now turn the run red instead of passing silently:

- if the name list is empty, the nightly sweep is skipped and the run
fails, but abandoned promotion candidates are still handled, because
their protection doesn't depend on that list
- if a package's versions can't be read, that package is skipped and the
run fails at the end
- if a DELETE is refused, `hack/retention-delete.sh` logs an error with
the HTTP status, goes on with the remaining versions and fails the run
(before, this was only a warning)
- if names resolve but none of the packages is linked to this
repository, nothing gets swept, so the run fails too, after the
promotion sweep has run (that sweep itself is skipped only when
candidate protection can't be resolved)

Promotion-candidate protection itself is unchanged. The contract test
that pinned the old org listing now pins the new behaviour, and both new
scripts have bats tests.

Known gap: a component that disappears from the nightly tree drops off
the list, so its old dated nightlies are never pruned. Today none of the
packages outside the list has more dated nightlies than the keep window.

Dry-run on this branch is green:
https://github.com/cozystack/cozystack/actions/runs/38075159900. It
resolves 59 packages, skips `cozystack/cozystack-ui` with the warning,
and would delete 418 old nightly versions plus one abandoned promotion
candidate. Deleting needs a different token. The app token lists
versions, but a one-version apply run
(https://github.com/cozystack/cozystack/actions/runs/37939905343) got
`404 Package not found` on DELETE for a version it had just listed. The
same DELETE with this repository's `GITHUB_TOKEN` and the job's
`packages: write` worked in a second run
(https://github.com/cozystack/cozystack/actions/runs/37941084456), with
no change to package settings. So `hack/retention-delete.sh` now deletes
with `GITHUB_TOKEN`, and listing stays on the app token.

That delete was shown on a package linked to this repository. One of the
swept packages, `cozystack/cozystack-ui`, is linked to another
repository, and I didn't try deleting there. Such a package is skipped
before its versions are read and named in a warning, so it can't turn
every apply run red. The follow-up for it is in #3278.

`RETENTION_APPLY` stays off, so the scheduled run is a green dry-run
until someone turns it on.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

N/A

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(retention): the GHCR retention workflow takes package names from the published nightly tree instead of the org package listing, which rejects the app token, and fails the run whenever names, versions or deletions cannot be resolved.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Nightly package retention now targets packages referenced by the
published nightly tree and keeps the 14 newest eligible versions per
package.
* Promotion candidates older than 30 days are eligible for pruning only
when their digests are not protected by qualifying release pull requests
or branch tips.
* **Bug Fixes**
* Packages linked to another repository are skipped, and sweep failures
are reported without preventing other eligible packages from being
processed.
* **Documentation**
* Clarified which nightly versions and promotion candidates qualify for
retention.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 12:02:35 +05:00
myasnikovdaniil 34488db766 ci(e2e): never tear down the Proxmox tenant on the runner's default context (#4807) 
## What this PR does

The Proxmox e2e teardown could delete from the wrong cluster. It runs
under `always()` and got its kubeconfig only from `GITHUB_ENV`, which
the "Stage credentials" step writes. If a step before that fails, for
example a missing stand variable, `KUBECONFIG` is never set. The three
`kubectl delete` calls then go to whatever cluster the self-hosted
runner's own kubeconfig selects.

Every teardown call now passes the staged file with `--kubeconfig`.
Stage fails unless that file's current context resolves to a server, and
prints kubectl's own error when it does not. Before, an empty
`COZY_KUBECONFIG` secret let client-go fall back to its defaults,
in-cluster credentials among them. The suite step gets the staged path
in its own `env`. The `GITHUB_ENV` export is gone, so no step in this
workflow picks its cluster up implicitly.

Tear down decides whether to skip from how Stage ended, not from the
file. If Stage did not succeed, the suite never ran and there is no
tenant to remove, so the step prints a notice and exits 0. If Stage
succeeded and the file no longer resolves, the step fails with kubectl's
error, because the tenant and its VMs may still be there. A side effect:
a run whose Stage fails no longer cleans up a tenant left by an earlier
`keep_tenant` run. The next run that passes Stage without `keep_tenant`
removes it.

The Stage and Tear down scripts now have unit tests in
`hack/proxmox-teardown-contract.bats`, run against a stub `kubectl`. I
also ran both against a real kubectl v1.37.1 for every Stage outcome.
The files were missing, empty, garbage YAML, without a current context,
with a dangling context, without a server, and valid. One more case was
a runner without kubectl.

The deletes also no longer stop at the first failure. Under `set -e`, a
failed first delete skipped the tenant delete, and the tenant's VMs
stayed on the hypervisor. Now each delete runs, and the step fails at
the end if any of them failed. With a recording `kubectl` that fails one
chosen delete, all three ran in every case and the step exited 1. With
the change reverted, a failing first delete was the only one that ran.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(e2e): the Proxmox e2e teardown always uses the staged kubeconfig, so it can no longer delete from the runner's default cluster. It skips only when Stage did not succeed and fails when the kubeconfig breaks after Stage. A failed delete no longer skips the tenant delete.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Improved reliability of automated Proxmox test setup and cleanup.
Tests now stop setup when Kubernetes configuration is unavailable, skip
cleanup when setup fails, and attempt all resource deletions before
reporting cleanup failures.
* **Tests**
* Added checks for configuration validation, cleanup behavior, and
handling of failed deletion attempts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 12:02:25 +05:00
myasnikovdaniil 2db18a0c86 test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport (#4657) 
## What this PR does

The kubernetes suites' LINSTOR cleanup check now compares dataset names
per node, and prints the leftover names when it fails (#4651). This PR
puts the rest of what a triager needs into `cozyreport.tgz` on every
run, green or red: the size of each dataset, its `origin` and
`creation`, the snapshots, the pool's pending frees, and the LINSTOR
volume list.

What `hack/cozyreport.sh` now collects:

- `linstor/zfs-pools.txt`: the zpool behind each ZFS storage pool, per
node, taken from `StorDriver/StorPoolName` in `linstor
--machine-readable sp l`.
- `linstor/zfs/<node>.txt`: that node's satellite runs `zpool get -p
freeing,leaked,allocated,free` and `zfs list -p -t all -o
name,used,avail,refer,origin,creation -r` over the node's pools.
- `linstor/volumes.txt`: `linstor v l`, next to the existing
`resources.txt`.

Sizes are exact bytes, so the sizes of leftover datasets can be summed
against the pool root's `avail`. That `avail` can differ from
`storage-pools.txt` without anything leaking: on a node with only thick
pools, `linstor sp l` serves the free capacity cached at the satellite's
last report. A dataset LINSTOR could not destroy because snapshots
depend on it is renamed to `_deleted_<name>` in the same pool, so a
search for leftovers has to look for that prefix as well as `pvc-`.

A pool that `zpool get` or `zfs list` cannot open does not cost the
listing of the other pools. `zpool get`'s message and exit status go
into the listing, and `zfs list` lists the rest. A node LINSTOR names no
ZFS pool for gets a note instead of a listing, because `zfs list -r`
with no pool argument lists every pool on the machine. A satellite whose
`spec.nodeName` cannot be read gets a note that says so. All reads go
through the bounded exec reader, like the other LINSTOR reads, and the
time budget in the script header counts them.

The second commit changes how that reader marks a read that wrote output
and then exited non-zero without hitting the timeout. It used to append
`TRUNCATED ... part way through, not because the command finished`. But
`kubectl exec` passes the remote command's status through, and `zfs
list` over one missing pool prints a complete listing of the others and
exits 1. The marker now names the exit status and says the status can't
tell an error after complete output from a cut stream. This changes the
marker for every LINSTOR exec read. The timeout marker stays as it was.

Tested with `hack/cozytest.sh hack/cozyreport.bats`. The new tests run
the controller and satellite shells against stub `linstor`, `zpool` and
`zfs`, with the real `jq`. They cover a failed and an unparsable
storage-pool read, a pool row with a missing, empty or blank backing
name, a non-ZFS pool, a pool name with a space, a pool that `zpool` and
`zfs` cannot open, a failing `zfs`, and an unreadable node name. A test
goes red if any of these break: the filter that keeps non-ZFS pools out,
the filter that drops a missing, empty or blank pool name, the jq exit
status, the `zpool` fallthrough, the node-name note, or the reader's new
wording.

Refs #4653

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): cozyreport records the ZFS datasets of each node's LINSTOR pools in exact bytes, and the LINSTOR volume list, on every e2e run.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Collection reports now include LINSTOR volumes, ZFS pool capacity and
usage, and per-node ZFS datasets and snapshots with exact-byte details.
Dataset listings are scoped to the ZFS pools LINSTOR reports for each
node.
* Reports include notes or warnings when node, pool, or dataset
information is missing or cannot be collected.

* **Documentation**
* Clarified storage report contents, artifact markers, bounded reads,
and how unsupported data, incomplete results, collection limits, and
per-object warnings are reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 12:00:15 +05:00
Aleksei Sviridkin 95c2d85b5b fix(etcd): leave compaction to etcd and pass the real quota to defrag (#4873) 
## What this PR does

The hourly defrag CronJob ran `etcd-defrag` with its defaults, and two
of them do not fit the cluster the chart creates.

- `--compaction` (default `true`) compacts to the latest revision before
every defrag. The EtcdCluster already compacts itself
(`autoCompactionMode: periodic`, `autoCompactionRetention: 5m`), so the
extra compaction either fails with `mvcc: required revision has been
compacted` when nothing was written since the last auto-compaction,
which is what every defrag Job log shows on our clusters, or, on a busy
cluster, drops the history the retention keeps for watchers to resume
from.
- `--etcd-storage-quota-bytes` defaults to 2 GiB, so `dbQuotaUsage` in
the defrag rule is measured against 2 GiB rather than the
`quotaBackendBytes` the chart sets (95% of `size`, about 3.8 GiB for the
default `4Gi`).

The CronJob now passes `--compaction=false` and the chart's own quota
through the existing `calculateQuotaBackendBytes` helper. The new unit
tests check both arguments and that the defrag quota equals the
EtcdCluster's `quotaBackendBytes` for a non-default size. Without the
template change all three fail.

The same templates are on `release-1.6` and the change applies there
cleanly.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff is two template arguments and a unit test in
`packages/extra/etcd`, with no change to values, schema, defaults or
package names, so nothing in the trigger map applies.

### Release note

```release-note
fix(etcd): the defrag job no longer compacts on top of etcd's periodic auto-compaction, and it evaluates its 80% quota threshold against the cluster's real quotaBackendBytes instead of 2 GiB.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Improvements**
* The etcd defragmentation job now uses the cluster’s configured backend
storage quota and skips compaction before defragmenting. This keeps the
defragmentation operation aligned with the cluster’s storage settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-12 08:57:34 +03:00
Andrey Kolkov 770f515105 docs(backupstrategy-controller): state the first-commit pin exception 
docs/agents/image-refs.md forbade committing a bare tag without saying
how a new image's first pin can carry a digest it does not have yet.
Record the exception the backup client keys rely on: the bare tag is a
placeholder the producing `make image` replaces, and an unbuilt tree
pulls a tag that does not exist. Drop the securitygroup-controller
example, which is pinned by digest now.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-10-12 09:54:52 +04:00
Andrey Kolkov 647407545d fix(backupstrategy-controller): run backup clients on a package-owned image 
The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB
image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the
migration runner, and whose pins were stamped from packages/core/platform
so every migration change moved the backup clients too.

Build images/backup-http-client (alpine + curl + jq + ca-certificates,
non-root) from this package's Makefile and stamp both client keys from
it. The cross-package stamp in packages/core/platform/Makefile and the
platform-migrations-specific check in hack/image-pin-consistency.bats go
away; the general one-digest-per-repository check still covers the two
keys sharing one image.

Fixes #3420

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-10-12 09:54:52 +04:00
Yasar Rahman 8be7fb2e66 feat(redis): expose min-replicas-to-write and min-replicas-max-lag 
Lets an operator bound the writes lost when an isolated Redis master is
failed over: the master refuses writes while fewer replicas than
configured are connected and recent. Both values are unset by default, so
the rendered Redis config is unchanged.

Redis treats 0 in either directive as disabling the check, so the
chart refuses minReplicasMaxLag 0 next to a replica requirement
instead of silently turning the protection off.

Negative values are refused by the schema (minimum 0), since Redis
rejects them at CONFIG SET and the operator would stop healing the
Sentinels of that instance. A requirement of replicas or more is
refused at render: a master with replicas pods has replicas-1
replicas, so it would refuse every write for good.

Assisted-by: LLM
Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
 2026-10-12 02:03:17 +05:00
Aleksei Sviridkin 33ca8b67bb feat(redis): run Sentinel on the same image as the Redis pods (#4831) 
## What this PR does

The Redis application pins its Sentinels to `redis:6.2.6-alpine`
(`packages/apps/redis/templates/redisfailover.yaml`), while the data
pods run the version selected by `version` (`redis:8.4.0` for `v8`,
`redis:7.4.7` for `v7`). The pin was added in fde96dd7d so that the
freshworks-oss operator's new compiled-in default (`redis:7.2.4-alpine`)
could not move the Sentinels silently on upgrade, and its comment asks
for the bump to be made as a deliberate, separate change. This is that
change.

Redis 6.2 is end of life, and today every instance runs a Sentinel one
or two major versions behind the Redis it monitors. With this change
`spec.sentinel.image` is rendered from the same `files/versions.yaml`
entry as `spec.redis.image`, so a `v8` instance runs Redis and Sentinel
on `redis:8.4.0`, and a `v7` instance on `redis:7.4.7`. Nodes then pull
one engine image per instance instead of two.

The rollback concern behind the original pin was dump.rdb compatibility.
It does not apply to Sentinel: a Sentinel pod stores no RDB or AOF, only
a `sentinel.conf` on an `emptyDir` that the operator's
`sentinel-config-copy` init container rewrites from the operator
ConfigMap on every pod start. Rolling back the chart therefore starts
the older Sentinel image from a clean config. The image stays explicit
in the rendered RedisFailover, so an operator upgrade still cannot move
it.

Changes:

- `templates/redisfailover.yaml`: `spec.sentinel.image` uses
`redis.versionMap`, with the comment rewritten to say why it is pinned
and why a rollback is safe.
- `values.yaml` and `hack/update-versions.sh`: the `version` description
says that Redis and Sentinel run the same image (`update-versions.sh`
rewrites that line on `make update`, so it carries the same text).
`README.md`, `values.schema.json`, `api/apps/v1alpha1/redis/types.go`
and `packages/system/redis-rd/cozyrds/redis.yaml` are the `make
generate` output for that description.
- `tests/sentinel_image_test.yaml`: new helm-unittest suite; it asserts
that the Sentinel and Redis images share the major version for `v8` and
`v7`. The patterns pin the major version only, so a patch bump in
`files/versions.yaml` needs no test change.
- `hack/e2e-chainsaw/redis/chainsaw-test.yaml`: comment only. It
described the Sentinel container's `timeout` as busybox (exit 143); the
Sentinel now runs the Debian-based image with coreutils (exit 124). The
checks only read "non-zero", so behaviour is unchanged.

Upgrade behaviour: on the first reconcile after the upgrade, every Redis
instance rolls its Sentinel Deployment once (the operator does not set a
strategy, so the default `maxSurge: 1`, `maxUnavailable: 0` applies to 3
replicas). A new Sentinel pod starts from the initial config, the
operator points it at the current master (`NewSentinelMonitor`, which
also sets `auth-pass`), and only then does it pass the readiness probe
that lets the next old pod go. The data pods are not touched.

Not changed: Harbor's Redis
(`packages/system/harbor/templates/redis.yaml`) keeps both images on
`redis:6.2.6-alpine`. Its data pods are PVC-backed, so the rollback
concern still applies there and it needs its own decision.

Testing done:

- `make generate` in `packages/apps/redis`: regenerated files committed;
a second run leaves no diff.
- `make test` in `packages/apps/redis` (helm-unittest v1.1.2): 50
passed. With the template change reverted, the 2 new tests fail and the
other 48 pass.
- `helm template` with `version=v8` and `version=v7`:
`spec.sentinel.image` and `spec.redis.image` are both `redis:8.4.0` and
both `redis:7.4.7` respectively.
- `helm lint` reports the same two errors as on `main` (icon URL,
`cozy-lib` not declared as a dependency), so nothing new.
- I checked the operator's Sentinel command set (`SENTINEL
REMOVE/MONITOR/SET auth-pass/SET
down-after-milliseconds/RESET/CKQUORUM/MASTER`, `INFO sentinel`) against
a Redis 8.6.2 Sentinel locally. It behaves the same as on 6.2. I have
not run the e2e suite. On a Cozystack v1.6.4 cluster we run Redis 8.4.0
data pods with 6.2.6 Sentinels today, but not this exact chart.

Backwards compatibility: no API change. Existing instances roll their
Sentinels once, as described above. Rolling the chart back rolls the
Sentinels back to `redis:6.2.6-alpine` with a fresh config.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Trigger map walked against the diff: the only user-visible text change
is the `version` description in `values.yaml`/`README.md`. `redis` is in
the website `Makefile` `APPS` list, so the release bot regenerates its
page. No field, enum, default or name changes, so
terraform-provider-cozystack is not affected.

### Release note

```release-note
feat(redis): Sentinel pods now run the same Redis image as the data pods (redis:8.4.0 for v8, redis:7.4.7 for v7) instead of the end-of-life redis:6.2.6-alpine. Existing instances roll their Sentinels once on upgrade; data pods are not restarted.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Redis Sentinel now uses the image selected by the Redis version
setting, keeping Sentinel and Redis pods on the same major-version
image.
* **Tests**
* Added checks confirming Sentinel and Redis images match with the
default version and when Redis 7 is selected.
* **Documentation**
* Clarified that Redis and Sentinel pods use the same image when
choosing a Redis version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-11 22:26:21 +03:00
Yan Bondarenko fe59a6bf91 fix(etcd): leave compaction to etcd and pass the real quota to defrag 
The hourly defrag CronJob runs etcd-defrag with its defaults, which compact
to the latest revision before defragmenting. Every cluster already compacts
itself with a periodic five-minute retention, so the second compaction
either fails with "required revision has been compacted" when nothing was
written since, or drops the history that retention keeps for watchers to
resume from.

The same defaults evaluate dbQuotaUsage in the defrag rule against 2 GiB
rather than the quotaBackendBytes the cluster runs with, so the 80 percent
threshold fires at a different database size than intended.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
 2026-10-11 20:21:26 +03:00
Aleksei Sviridkin 9a348eeb50 fix(kubernetes): delete orphaned resources in new post-delete job (#3856) 
## What this PR does
When tenant deletes the kubernetes cluster, 3 resources are orphaned and
leftover in the tenant namespace.
- jobs/<name>-oidc-bootstrap
- secrets/<name>-talos-ca
- secrets/<name>-talos-tls-cert

This PR adds a new "step 7" in existing
`packages/apps/kubernetes/templates/delete.yaml` in order to delete
these orphaned resources manually.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(kubernetes): Fixed the `<name>-talos-ca` and `<name>-talos-tls-cert` Secrets and the `<name>-oidc-bootstrap` Job being left in the tenant namespace after a Kubernetes cluster is deleted.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Kubernetes release cleanup now removes the Talos CA and TLS
certificates before their associated Secrets, and also removes the OIDC
bootstrap Job.
* The final cleanup report now includes best-effort failures from Step
7, alongside failures from Steps 3, 5, and 6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-11 20:08:27 +03:00
Aleksei Sviridkin 84e5e949fe test(ci): pin the label-writing workflows with contract tests (#3664) 
Closes #3589

Adds `hack/label-workflows-contract.bats`, following the pattern of
`hack/promote-gate-contract.bats` / `hack/release-freeze-contract.bats`
(same helpers: comment-stripped pins over job/input blocks, one filter
per language since `pr-labeler.yaml` mixes YAML `#` and JavaScript `//`
comments).

## What it pins

**Cross-file label resolution** (the headline check from the issue):
- every value in `typeToKind` and `scopeToArea`, plus the three literal
`toAdd.add(...)` labels (`area/release`, `kind/breaking-change`,
`area/uncategorized`), resolves against a `name:` in
`.github/labels.yml`
- `stale.yaml`'s four label inputs and all 16 exempt entries (12 issue +
4 PR) resolve the same way; failures name the missing label

**Structure:**
- trigger list stays `[opened, reopened, synchronize]` — `edited` cannot
creep back in as an executable line
- exactly the five documented type mappings; a mapping demoted to a `//`
comment stops counting
- additive-only: `addLabels` once, no `removeLabel`/`setLabels`,
`!existing.has(l)` filter present, `area/uncategorized` fallback present
- stale policy: label choices, exempt-list sizes, `days-before-stale:
60`, `days-before-close: 14`, `operations-per-run: 100`,
`remove-stale-when-updated: true`, and the specific renewal-path
exemptions (`lifecycle/frozen`, `triage/accepted`, `do-not-merge/hold`)

## Verification

- Passes under both `bats` and `hack/cozytest.sh` (POSIX-only constructs
— no process substitution; the first draft used one and cozytest.sh
caught it, which is a nice validation of running both)
- Mutation-checked per the issue's bar — each of these went red and was
restored: typo in a `scopeToArea` value, `edited` added to the trigger
list, fallback line demoted to `//`, `days-before-stale` changed, typo
in an exempt label, `feat` mapping demoted to `//`, typo in the literal
`kind/breaking-change`
- Picked up automatically by `BATS_UNIT_FILES` (wildcard) — no Makefile
change
- The two documented drift examples (the `edited` event claim and the
`revert` type in `docs/agents/contributing.md`) are left as-is: they are
doc statements, and which side is authoritative is a maintainer call.
The tests pin the workflow's current executable behavior.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added automated checks for pull request labeling and stale-item
workflows.
* Validates workflow labels against defined labels and pull request type
mappings against the contributing guide, including documented unmapped
types.
  * Checks pull request event triggers and label-writing behavior.
* Verifies stale-item labels, exemptions, policy values, operation
limits, and scheduled and manual triggers, including the absence of
user-activity triggers.
* Ensures commented-out workflow declarations do not satisfy these
checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-11 19:43:10 +03:00
Aleksei Sviridkin 76a4af95c0 docs(monitoring): correct the release name the OIDC comment, guide and test fixture encode (#3468) 
## What this PR does

The Grafana OIDC `clientId`, audience-scope name, client-secret name and
users-Job name are all derived from the Monitoring release name. In a
running cluster that release is always `monitoring-system`: the tenant
chart's outer HelmRelease `monitoring` points at `extra/monitoring`,
whose `templates/helmrelease.yaml` creates the inner HelmRelease `{{
.Release.Name }}-system` that carries `system/monitoring` and renders
OIDC. There is no release named `monitoring` that renders OIDC.

Three places encoded a release named `monitoring` (i.e.
`<ns>-monitoring`, where a cluster renders `<ns>-monitoring-system`),
per #3338:

- `packages/system/monitoring/templates/_helpers.tpl` header comment —
both example lines were wrong. The per-tenant release is
`monitoring-system`, and the platform Grafana lives in the `tenant-root`
namespace, not `cozy-monitoring`. (`cozy-monitoring` only hosts the
ExternalName services that redirect to the platform stack the root
tenant hosts in `tenant-root` — see
`packages/system/cozystack-basics/templates/monitoring-external-services.yaml`.)
- `docs/oidc-grafana.md` — placed the platform release in
`cozy-monitoring`; it lives in `tenant-root`. It also named the
per-tenant release `monitoring`; that is `monitoring-system` too, so
both instances now read `monitoring-system` and differ only by
namespace.
- `packages/system/monitoring/tests/oidc_test.yaml` — the suite pinned
`release.name: monitoring`, so its clientId / audience-scope /
client-secret / users-Job assertions pinned names no cluster renders.
Both fixtures are set to `monitoring-system` and the derived expected
values updated to what a cluster actually renders.

Nothing changes at runtime — the templates are self-consistent whatever
the release is called; this only aligns the descriptions and the test
fixture with reality.

Out of scope here (from the issue, left for a maintainer decision):
- Confirming/removing the apparently-dead `cozystack.monitoring`
PackageSource in `packages/core/platform/sources/monitoring.yaml` (issue
item 3) — removing a source is a call I can't safely make.
- Re-checking Detail B of #3259 against the corrected topology (issue
item 4).

Verification, rebased on current main: `helm unittest -f
'tests/oidc_test.yaml' packages/system/monitoring` — 39/39 pass; the
whole monitoring chart passes as well. After setting the fixtures I
updated only the assertions helm-unittest reported as newly diverging.

Disclosure: prepared with AI assistance; I reviewed the change and
verified the release-name topology against the source files cited above.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. The three changed paths are a `system/monitoring`
chart `_helpers.tpl` comment, the repo-internal `docs/oidc-grafana.md`,
and a chart test fixture. None match a listed trigger: no change under
`packages/apps/` or `packages/extra/`, no
`packages/core/platform/values.yaml`, and no
variant/bundle/component/release-asset/Talos/tooling change. So no
downstream repository is mechanically affected. (The website's own OIDC
page referenced in cozystack/website#597 is a separate artifact the
reporter is already engaged with, outside this repo's trigger map.)

- [x] No downstream repository is affected by this change

### Release note

```release-note
docs(monitoring): correct the release name (`monitoring-system`) and platform namespace (`tenant-root`) encoded in the Grafana OIDC helper comment, the OIDC guide, and the chart test fixture.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
- Clarified that both platform and tenant Grafana instances use the
`monitoring-system` release name. The platform instance runs in
`tenant-root`, while tenant instances run in their respective tenant
namespaces.
- Updated the OIDC integration guidance to reflect the platform
namespace and tenant release naming.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-11 19:40:30 +03:00
Aleksei Sviridkin 4295930989 feat(dashboard): show categories brought by tapped repositories in the sidebars (#4741) 
## What this PR does

The marketplace and console sidebars render a fixed category list,
`["IaaS", "PaaS", "NaaS"]`, and drop every other
`spec.dashboard.category`. A repository tapped with `cozypkg` that
brings its own category is reachable only at the bottom of the full
application list, and its installed applications get no console section
at all.

Both sidebars now follow `groupByCategory`, the order the marketplace
page already uses: IaaS, PaaS, NaaS first, then any other category
alphabetically. `Administration` stays in the Admin portal, as before.
Unknown categories get the generic grid icon.

The note in `docs/agents/contributing.md` that described the hardcoded
list is removed, since it no longer holds.

Tests: two new cases in `sidebar-sections.test.tsx` (a tapped
`Paleocomputing` category appears after the built-in ones in both
sidebars, `Administration` stays out). Both fail on the previous code.
The console suite (448 tests), `pnpm typecheck` and `eslint` on the
changed files pass.

### Screenshots

Same cluster and tenant, a repository tapped that brings the
`Paleocomputing` category.

Before, marketplace: the category is only reachable at the bottom of the
full list.

![before
marketplace](https://raw.githubusercontent.com/tym83/cozystack/pr-assets/dashboard-categories/before-marketplace.png)

After, marketplace: the category gets its own filter.

![after
marketplace](https://raw.githubusercontent.com/tym83/cozystack/pr-assets/dashboard-categories/after-marketplace.png)

Before, console: no section for the tapped applications.

![before
console](https://raw.githubusercontent.com/tym83/cozystack/pr-assets/dashboard-categories/before-console.png)

After, console: installed applications of that category get their own
section.

![after
console](https://raw.githubusercontent.com/tym83/cozystack/pr-assets/dashboard-categories/after-console.png)

### Downstream repositories

Walked the trigger map against the diff: it touches only the console
sidebar code and a note in `docs/agents/contributing.md`. No package,
values schema, CRD, installer value or `ApplicationDefinition` field
changes. The website page on `ApplicationDefinition` does not restrict
`spec.dashboard.category` values, and `external-apps-example` uses
`PaaS`, which keeps working as before.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(dashboard): categories brought by tapped repositories now appear in the marketplace and console sidebars, after IaaS, PaaS and NaaS, instead of only in the full application list.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Marketplace and Console sidebars now show application categories
beyond the built-in options, including custom categories. Categories
appear in the order provided by the application grouping.
  * The Administration category remains excluded from both sidebars.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-10-11 18:56:38 +03:00
Aleksei Sviridkin be55f6eb2b test(ci): load the shared bats helper in the labeler contract test 
Every unit bats file under hack/ must load test_helper so it runs with
nounset, and hack/bats-strict-setup.bats fails the suite for a file
that does not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-10-11 18:42:11 +03:00
shaurya703 79e95a1b30 test(ci): pin the label-writing workflows with contract tests 
pr-labeler.yaml and stale.yaml decide which labels land on issues and
PRs, and neither had a test behind it. A mistyped label name is not an
error, it is a rule that silently stops matching.

hack/label-workflows-contract.bats pins:

- every label either workflow can write or match resolves against a
  name: in .github/labels.yml (typeToKind, scopeToArea, the three
  literal adds, stale/close labels, both exempt lists)
- the typeToKind mappings are exactly the rows of the type table in
  docs/agents/contributing.md, in both directions, and a type the table
  lists under "(no kind label)" has no key in the workflow
- the trigger list stays [opened, reopened, synchronize] and 'edited'
  does not come back in either YAML spelling
- the labeler stays additive-only (addLabels only, no removeLabel or
  setLabels) with the area/uncategorized fallback
- stale policy values: label choices, 12 issue and 4 PR exemptions,
  days-before-stale 60, days-before-close 14, operations-per-run 100
- stale runs from its daily cron and workflow_dispatch only

JS // and YAML # comments are stripped before matching, so a mapping or
a gate demoted to a comment cannot satisfy the contract. Negative checks
use the grep -c / -eq 0 form of the sibling contract files: a negated
pipeline is ignored by set -e, so it cannot fail a test under
hack/cozytest.sh and fails under bats only as the last command.

Every assertion was mutation-checked under both bats and
hack/cozytest.sh. POSIX-only constructs; picked up automatically by
BATS_UNIT_FILES.

Closes #3589

Signed-off-by: shaurya703 <kesar.shaurya@gmail.com>
 2026-10-11 18:40:02 +03:00