Commit Graph
2485 Commits
Author SHA1 Message Date
Mattia Eleuteri e510d49b3d feat(monitoring): re-render node agents when a monitoring label changes
The node fluent-bit builds its tenant routes from a lookup of namespace
labels, and helm-controller upgrades a release only when its chart or
values digest changes. A tenant turning monitoring on therefore got no
route, and a removed one kept its route, until an unrelated change
upgraded monitoring-agents.

The reconciler digests the label across all namespaces and requests a
forced upgrade of the release when the digest moves. It digests every
labelled namespace rather than repeating the chart's selection, so it
cannot drift from the chart; the cost is an occasional upgrade that
renders the same config.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:55 +02:00
Mattia Eleuteri cdff5ed380 feat(monitoring): route tenant logs to the tenant's own VictoriaLogs
A tenant that runs its own monitoring stack gets a VictoriaLogs instance,
but the node fluent-bit sends every container log to global.target, so
that instance stays empty. The tenant chart already names the stack a
namespace reports to in namespace.cozystack.io/monitoring, the label the
per-tenant vmagent selects on, so the log path keys off the same label.

The routed record leaves the default outputs (keep false): a tenant with
its own monitoring holds its logs, the root store no longer does. The
re-tagged record re-enters the filter chain, where the Match * "Add
tenant" cannot overwrite the key, so a per-target Set labels it with the
tenant it is sent to. One rewrite_tag carries every rule, keeping a
single emitter however many tenants are routed.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:54 +02:00
Aleksei Sviridkin 8b861668b8 ci(release): build pre-release images for amd64 and arm64 and gate releases on it (#4505)
## What this PR does

Pre-release builds now publish every image as an amd64 and arm64 index,
and a release fails if anything it ships is single-arch. Main, release
lines and pull requests keep building amd64 only. It builds on #4498,
which made every image buildable for both architectures. Part of #1961.

**This PR is on hold until everything under "Merge after" has landed.**
Merged earlier, the new gate fails the next rc on the images those PRs
fix.

### How a pre-release is built

A pre-release tag gets a second job, for arm64, next to the existing
release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs
on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same
images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`.
`prepare-release` waits for it, and if the arm64 job fails the rc fails
with it. There is no amd64-only fallback. Stable tags do not run it:
promotion copies the rc digests with `skopeo copy --multi-arch all`, so
a stable release inherits the indexes.

After the amd64 build, the stitch joins each amd64 image with its arm64
twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag
of the amd64 image moves to the index, the component-versioned ones
included. The script then rewrites the digests in the tree and
republishes the packages artifact and the installer chart pinned on
them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an
old digest survives anywhere outside `charts/`.

### The gate

A new check fails the rc when a pinned digest in the tree is not an
index with both linux/amd64 and linux/arm64. It lives in
`hack/verify-multiarch.sh`. It checks first-party and third-party refs
alike. Images that are amd64 by nature go into
`hack/multiarch-allowlist`, one repository per line, each with a
mandatory reason. Today that is only the e2e sandbox. An entry that
matches nothing is reported as stale.

The static gate only sees digest-pinned refs in the tree. A tag-only
ref, an image from a vendored chart default, or one an operator starts
at runtime is invisible to it. So the rc e2e also audits every image its
nodes pulled, with the same check and the same allowlist. Neither covers
a package the e2e never installs.

### Nightly

A new nightly workflow builds main for arm64. It keeps the arm64 build
cache warm, since the rc job only reads it. It also prints the
single-arch refs that would block the next rc. Nothing it builds is
published.

### Smaller changes

- `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so
the arm64 build keeps a cache of its own.
- `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the
amd64-only e2e sandbox.
- The kamaji provider image is pushed under the build's `IMAGE_TAG` like
every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent
builds overwrote each other's component tag. Closes #4503.
- `hack/nightly-mirror.sh` verifies a copy against the raw manifest
digest. Plain `skopeo inspect` resolves an index to one platform's
child.
- matchbox is rebuilt for both platforms in the amd64 release job. The
arm64 leg skips the talos package, so there is no arm64 half to stitch,
and its Dockerfile only copies files, so no QEMU is needed. Releases
then network-boot arm64 machines too. Closes #4524.

The stitch also rewrites the kamaji ref inside `files/components.gz`,
recompressed with `gzip -n` so the bytes are reproducible, so the kamaji
control-plane provider ships multi-arch too. keda and kuberture now name
the repository next to their pinned digest, so the gate can resolve
them; their rendered manifests do not change.

### Merge after

- #4552 builds the Talos installer, matchbox, Harbor and the Velero
KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator,
kilo and migration-controller to the root `build:` list. It replaces
#4507, #4515, #4525 and #4528.
- #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and
keycloak-kms-proxy to their multi-arch releases, which are already
published.

The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and
#4522 here, and the multi-arch PRs in
ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler
and keycloak-kms-proxy.

### Verification

The bats suites pass: stitch, verify-multiarch, build-matrix,
common-envs, nightly-mirror and the release contracts. The first three
also pass under `hack/cozytest.sh` with dash. Each new test was red
before its implementation, and the verify-multiarch and stitch tests
each have a mutation that turns them red. I ran `verify-multiarch
--report` against main. It lists the first-party refs the stitch will
fix, plus the third-party and special cases above. The workflows have
not run yet. The arm64 job's tools and duration, the stitch against real
registries, and the audit are checked for the first time on the nightly
and on the next rc.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Nothing under `hack/` is moved or renamed, and no make target changes
its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The
satellite repositories listed under "Merge after" are prerequisites, not
follow-ups this change forces on them.

### Release note

```release-note
ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Release candidates now build amd64 and arm64 images in parallel and
combine eligible images into verified multi-architecture indexes before
publication.
* A nightly arm64 image build is available for testing and supplies a
build cache for release-candidate builds.
* **Bug Fixes**
* End-to-end checks audit whether pulled images support both
architectures. Audit failures block prerelease checks, while stable
releases continue with a warning.
* End-to-end test artifacts now include collected image references and
multi-architecture audit results.
* **Documentation**
* Updated release and image guidance covers multi-architecture builds,
verification checks, and troubleshooting, including arm64 build and
stitching failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 15:35:13 +03:00
Aleksei Sviridkin d9ed566326 fix(migration): restore the class reclaim policy on imported volumes (#4480)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The VM import handoff sets the transferred `PersistentVolume` to
`Retain` so the data survives the swap from the Forklift-populated claim
to the VMDisk's claim, and then never gives the policy back. Every
imported disk therefore ends up with `persistentVolumeReclaimPolicy:
Retain`, while its StorageClass says `Delete` and every VMDisk created
directly carries `Delete`. When the tenant later deletes an imported
VMDisk through the normal path, its DataVolume and claim go away, but
the volume stays `Released` with its backing storage: an orphaned,
billed volume that no tenant can see or remove.

#### How it was measured

On a lab cluster with a LINSTOR-backed `replicated` class
(`reclaimPolicy: Delete`): a 16 GiB VM imported from vSphere produced a
VMDisk whose PV was `Retain`, while every non-imported VMDisk's PV in
the cluster was `Delete`. Deleting the VMInstance and then the VMDisk
removed the DataVolume and the claim and left the PV `Released`, still
`Retain`, with its LINSTOR resource in place. Patching that PV back to
`Delete` made the CSI provisioner delete both the PV and the LINSTOR
resource, which confirms the reclaim policy is the only thing keeping
it.

#### The fix

`Retain` is now held only for the window it protects. A new step,
`settleReclaimPolicy`, restores the policy the volume's StorageClass
declares (`Delete` when the class names none or no longer exists) once
the handoff is provably over, and until then reports "not yet" and
touches nothing:

- the replacement claim is `Bound` and the PV is `Bound` with a
`claimRef` naming that claim by UID, as the PV controller reports it
rather than as the controller wrote it, so no release is in flight;
- the transferred claim is gone;
- the claim's one and only owner is the VMDisk's DataVolume, as a
controller reference whose UID matches the live DataVolume (CDI's
adoption), so the only deletion that can reach the volume is the
VMDisk's own;
- the VMDisk exists and carries this task's output markers.

`fulfill` runs this step after adopting every disk and before creating
the VMInstance and deleting the scaffolding VM, so a VM is reported
`Succeeded` only once its volumes carry their class's policy, and an
import that fails after this point still leaves disks that clean up like
any other. The step is a compare-and-set, so repeated passes and
restarts are no-ops once settled. It also runs on the path that recovers
outputs whose status was never written, for tasks caught mid-handoff by
an upgrade. The comment that called permanent `Retain` "part of the
contract" and the matching paragraph in
`packages/system/migration-controller/README.md` are updated; the README
also says how to fix volumes left on `Retain` by imports completed
before this change, since those tasks are terminal and are not
reconciled again.

In practice a VM now stays in `Creating` for one more requeue (about 20
s) after the swap, while the PV controller rebinds and CDI adopts the
claim.

#### Tests

- `TestSettleRestoresTheClassReclaimPolicy`: a finished handoff gets the
class's policy back: `Delete`, `Retain` when the class asks for it, and
`Delete` when the class names none or is gone.
- `TestSettleIsIdempotent`: a second pass over a settled volume reports
settled and does not rewrite the PV.
- `TestSettleKeepsRetainWhileTheHandoffIsOpen`: thirteen cases, each
breaking one condition (claim not created by this import, claim not
bound, PV still `Released`, `claimRef` still on the old claim,
`claimRef` on the new claim with a stale UID, transferred claim still
present, claim not yet adopted by CDI, an extra owner, a DataVolume
owner that is not the controller, an owner reference to a previous
DataVolume, DataVolume missing, VMDisk missing, VMDisk not created by
this import). Every one keeps `Retain`.
- `TestFulfillRestoresTheReclaimPolicyBeforeFinishing`: behavioural
regression over `fulfill`. The first pass swaps the volume, keeps
`Retain`, creates no VMInstance and keeps the scaffolding VM. After
simulating the PV controller and CDI, the second pass restores `Delete`,
creates the VMInstance and removes the scaffolding VM. With the new step
turned into a no-op, this test fails.
- `TestFulfillSettlesReclaimedOutputs`: the recovery path for outputs
whose status was never written. With the handoff still open the volume
keeps `Retain`; once it is finished the policy goes back to `Delete`.
- The existing `adoptVolume` tests still assert `Retain` during the
swap.

`go test ./internal/migrationcontroller/...` and `make test-controllers`
pass, `go vet` is clean, and `make generate` leaves the tree unchanged.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `internal/migrationcontroller/`,
`packages/system/migration-controller/README.md` and a comment in
`packages/system/migration-controller/templates/rbac.yaml`, which match
no entry of the trigger map; the website's VMware migration page does
not describe the reclaim policy, so it needs no change.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(migration): disks imported from VMware no longer keep a Retain reclaim policy after the import. The volume gets its StorageClass policy back once it is bound to its VMDisk, so deleting an imported VMDisk deletes its volume instead of leaving an orphaned Released PersistentVolume. Volumes imported before this fix keep Retain and have to be patched back by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Imported volumes are protected during claim transfers, then returned
to their StorageClass reclaim policy after the replacement claim and
volume are safely adopted. If the class or policy is unavailable, the
policy defaults to `Delete`.
* VM imports wait for volume adoption and reclaim-policy settlement
before completing, helping prevent premature volume deletion and
incomplete imports. Previously unrecorded handoff outputs are also
settled before being returned.
* **Documentation**
* Clarified reclaim-policy behavior during imports and how to restore
cleanup behavior for volumes from older imports.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 14:41:04 +03:00
Aleksei Sviridkin 717d091970 fix(backups, postgres): start a Postgres restore from the backup it names (#4443)
## What this PR does

A Postgres `RestoreJob` now starts from the base backup its `Backup`
refers to, and the PITR documentation describes the window the recovery
actually has.

**Starting from the named backup.** Without `recoveryTarget.backupID`
the barman-cloud plugin picks the base backup itself
(`internal/cnpgi/restore/restore.go` in plugin-barman-cloud, then
`FindBackupInfo` in barman-cloud `pkg/catalog`): the newest backup of
the catalog when no `recoveryTime` is set, otherwise the newest one
whose end is not after `recoveryTime`, on any timeline since no
`targetTLI` is passed. A `RestoreJob` naming an older `Backup` therefore
started from another one, and on an archive that holds sibling timelines
— left by in-place restores that archived back under the source
`serverName`, before restored clusters got a fresh `newServerName` — it
could start from another branch and bring back that branch's data.
`FindBackupInfo` gives `backupID` precedence over time-based selection,
and CNPG's webhook does not count it among the mutually exclusive
targets, so it combines with `targetTime`.

- on restore the CNPG driver reads barman's `backupId` and the backup's
`stoppedAt` from the `Backup`'s `cnpg.io/Backup` and sets the new chart
value `bootstrap.backupID`, rendered as
`bootstrap.recovery.recoveryTarget.backupID`;
- it does not pin for a `recoveryTime` earlier than one second past
`stoppedAt`: the named backup cannot serve an instant before it is
consistent, and `stoppedAt` is truncated to the second while the catalog
compares barman's end time to the microsecond. Those requests keep the
plugin's time-based choice, as today;
- it does not pin either when the `cnpg.io/Backup` is gone.
plugin-barman-cloud's retention deletes it together with the base backup
it describes (`deleteBackupsNotInCatalog`), and the driver deletes one
only with its artifact, so a missing one means the catalog has lost that
base backup: pinning it would purge the target and then fail the
recovery. The plugin's choice recovers to the same end of the archive,
and the `RestoreJob` gets a `BaseBackupGone` warning event. A base
backup deleted from the object store by other means is still pinned for
as long as its `cnpg.io/Backup` stays, and that restore fails after the
purge; plugin-barman-cloud deletes such a `cnpg.io/Backup` on its next
maintenance pass (`deleteBackupsNotInCatalog`, every
`retentionPolicyIntervalSeconds`, retention policy or not), but only
from the primary of the cluster that took it, under the same UID.
Telling it apart earlier would need reading the barman catalog, which
the driver does not do; the documentation says so, and that deleting the
`cnpg.io/Backup` by hand is the way out;
- a re-restore into the same target clears a `backupID` left by a
previous one.

**Documentation** (`docs/operations/backup-classes.md`, PostgreSQL
PITR):

- the upper bound of the window is the last transaction (commit or
abort) in the archived WAL, not the newest segment: recovery ends just
before the first transaction after the target, and a segment switch or a
base backup is not one. On a database at rest every instant after its
last write fails with `recovery ended before configured recovery target
was reached`, including "at/before the most recent completed backup",
which the page recommended as the safe choice; the state as of the last
write is obtained by omitting `recoveryTime`;
- aiming at a backup's `stoppedAt` misses that backup (the plugin keeps
only a backup whose end is not after the target), falling back to an
older one or to `no target backup found`; the page now says to aim one
second past it;
- when a base backup removed from the bucket outside retention is still
pinned, and how to unpin it;
- how to read the upper bound from the archive, what pinning the base
backup changes, and a section on archives with several timelines:
PostgreSQL follows `recovery_target_timeline: latest` from the backup's
timeline through the consecutively numbered history files it can find,
so a missing `.history` confines the replay to the backup's own timeline
and the bound is the last transaction on that timeline, while a targeted
timeline that does not descend from the backup stops recovery with
`requested timeline N is not a child of this server's history`.

**Tests.** `TestCNPGRestoreBackupID` (the pinning rule),
`TestReconcileCNPGRestore_PinsTheRequestedBackup` (drives a restore
through the purge and reads back the patched `Postgres` app, including a
`cnpg.io/Backup` removed by retention, with and without `recoveryTime`),
`TestBuildPostgresAppRestorePatch_ReplacesBackupID`, a deep-copy test
for the new `stoppedAt` field, and three `helm unittest` cases for
`recoveryTarget`. With the rule disabled, 4 of the rule cases and 3 of
the reconcile cases fail; with the template change removed, 2 chart
cases fail; the retention case fails against the first version of this
PR, which pinned an id recorded on the artifact; the deep-copy test
fails once `StoppedAt` is dropped from the generated `DeepCopyInto`. `go
test ./internal/backupcontroller/...` and `helm unittest
packages/apps/postgres` (139/139, after rebasing on main) pass. Not run:
the e2e suite (`hack/e2e-chainsaw`), and a restore against a live
cluster with this driver.

### Screenshots

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/716.
`content/en/docs/next/operations/services/backup-classes.md` (and
`v1.6/`) carries a copy of the corrected "The recoverable window" and
"Discovering the earliest / latest restorable time" sections; the window
and `stoppedAt` corrections apply to released versions too, the
`backupID` part only once this is released.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: none expected. The diff adds `bootstrap.backupID` to the
postgres `values.schema.json`, but the provider does not manage the
`bootstrap` block (`internal/provider/postgresql_schema.go`: "bootstrap
blocks use server defaults"), and the `RestoreJob` CRD is unchanged.
Left unticked for a maintainer to confirm.
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backups): a Postgres RestoreJob now starts from the base backup its Backup refers to (recoveryTarget.backupID) instead of whichever backup the barman-cloud plugin picks, which on an archive with several timelines could restore another branch's data; a recoveryTime before that backup's end keeps the previous time-based choice. The PITR documentation now bounds the recoverable window by the last archived transaction rather than the newest WAL segment.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Restores can target a specific base backup and optionally replay to a
requested recovery time. If no backup is specified, the newest eligible
backup is selected automatically.
* **Bug Fixes**
* Restore handling warns when the referenced base backup is unavailable
and allows the plugin to select a backup from the catalog.
* **Documentation**
* Clarified recovery-window boundaries, timestamp precision, idle
databases, timeline branches, and behavior when a requested target
cannot be reached or predates the earliest base backup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 14:26:49 +03:00
Aleksei Sviridkin c740031453 Merge remote-tracking branch 'origin/main' into ci/build-arm64-leg
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-29 11:20:27 +03:00
Mattia Eleuteri 497dd6fc4f fix(backups): start a Postgres restore from the backup it names
Without recoveryTarget.backupID the barman-cloud plugin chooses the
base backup itself: the newest one in the catalog when no recoveryTime
is set, and the newest one ending by recoveryTime otherwise, on any
timeline since no targetTLI is passed. A RestoreJob naming an older
Backup therefore started from another one, and on an archive holding
sibling timelines - left by in-place restores that archived back under
the source serverName - from another branch, bringing back that
branch's data.

The restore reads barman's backupId and the backup's stoppedAt from
the live cnpg.io/Backup behind the Backup, and passes the id through
the chart as bootstrap.recovery.recoveryTarget.backupID. The plugin
gives backupID precedence over time-based selection, and CNPG does not
count it among the mutually exclusive targets, so it combines with
targetTime.

Two cases keep the plugin's choice. A recoveryTime earlier than one
second past stoppedAt: the named backup cannot serve an instant before
it is consistent, and stoppedAt is truncated to the second while
barman compares its end time to the microsecond. And a cnpg.io/Backup
that is gone: plugin-barman-cloud's retention deletes it together with
the base backup it describes, and the driver deletes one only with its
artifact, so pinning its id would purge the target and then fail the
recovery on every retry. That restore records a BaseBackupGone warning
event instead.

A re-restore into the same target clears a backupID left by a previous
one.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:38:11 +02:00
Mattia Eleuteri c6a4933cf6 test(migration): cover the recovery settle and the handoff guards
Recovering outputs that no status recorded goes through fulfill's
vm == nil branch, which settles the reclaim policy of volumes an older
controller left on Retain. Nothing drove that branch, so the write path
could be removed with the suite still green. A fulfill test now runs it
with the handoff open and finished.

The guards that stop the controller from rewriting the policy of a
volume it did not produce, or before CDI controls the claim, get one
case each: a claim or VMDisk marked by another import, a non-controller
DataVolume owner reference, and a claimRef naming the replacement claim
under a stale UID.

The storageclasses RBAC comment now also names the reclaim policy read.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 09:37:25 +02:00
Kirill Ilin 5d02c94e71 chore(hetzner-robotlb): update robotlb to 0.1.0
The upstream project moved from the intreecom GitHub organization to
treetscom, so the chart and image are now pulled from the new location.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-09-29 10:50:55 +05:00
Aleksei Sviridkin 5e32ef35b0 feat(build): build Talos, Harbor, the Velero KubeVirt plugin and the unlisted images for arm64 (#4552)
## What this PR does

This brings together four approved PRs that give the remaining
first-party images an arm64 build. Each of them lost its approval when
#4498 merged and GitHub retargeted it to main, so I merged them into one
branch to review and approve once. Each original PR has the full
description and review history, and I'll close them in favour of this
one.

- #4507 builds the Talos installer and matchbox for arm64. Releases also
publish the arm64 ISO, disk images, kernel and initramfs.
- #4515 builds the KubeVirt Velero plugin in-tree for amd64 and arm64.
- #4525 adds flux-plunger, keycloak-operator, kilo and
migration-controller to the root `build:` list, so CI rebuilds them like
every other image. A test keeps any new image target from being left
out.
- #4528 rebuilds the Harbor v2.15.1 images for amd64 and arm64, because
upstream publishes v2.15 for amd64 only.

Release builds still run with `PLATFORM=linux/amd64`, except the Talos
installer, so these images ship arm64 only once #4505 adds the arm64
leg. This PR makes that leg buildable.

A few commits are new on top of the originals. The velero test no longer
pins the plugin's image string, because the release build stamps its own
rebuild over it. It now checks the plugin init container by repository
name. A promote-retag test comment now says what it guards, and the
image-refs doc no longer calls its fixed gaps open. The merge with main
keeps the Harbor portal in the list of stages whose output has no
architecture.

### Screenshots

Not a UI change.

### Downstream repositories

I left every box empty and opened nothing in other repositories. Two
repositories are touched, and neither follow-up is needed before this
merges:

- cozystack/website names the amd64 assets in
`content/en/docs/next/install/talos/iso.md` and
`install/providers/oracle-cloud.md`. The install pages could offer the
arm64 downloads once a release carries them. No asset is renamed, so the
current links keep working.
- cozystack/talm points at `ghcr.io/cozystack/cozystack/talos:<version>`
in `charts/cozystack/values.yaml`. Once a release publishes that tag as
a two-arch index, the same reference works on arm64 nodes, so nothing
there has to change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(talos): the Talos installer image is built for amd64 and arm64, with arm64 profiles that carry drbd and zfs, and the bootbox matchbox image network-boots arm64 machines as well as amd64 ones. Releases publish the arm64 Talos ISO, disk images, kernel and initramfs.
feat(velero): the KubeVirt Velero plugin is now built in-tree for amd64 and arm64, so the velero server starts on arm64 clusters.
build: flux-plunger, keycloak-operator, kilo and migration-controller are now rebuilt by CI like every other first-party image, so their pinned images pick up source patches and base-image updates and are published for amd64 and arm64.
feat(harbor): the Harbor application now runs on arm64 nodes. Its component images are rebuilt from the pinned upstream v2.15.1 release for amd64 and arm64, because upstream publishes v2.15 images for amd64 only.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Talos images and boot assets now support both AMD64 and ARM64,
including architecture-specific installation and cloud images.
* Added build support for Harbor components and the KubeVirt Velero
plugin.
  * Nightly builds now include ARM64 release assets.
* **Improvements**
* Harbor uses pinned chart and image versions, with built image
references recorded automatically.
* Added checks to catch image packages missing from the main build and
to verify multi-architecture installer output.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 01:37:52 +03:00
IvanHunters 689b7eefb9 feat(system): update ingress-nginx, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy to multi-arch releases (#4549)
## What this PR does

Four components move to their first releases built for both linux/amd64
and linux/arm64. The current pins are amd64 only, so these pods cannot
start on an arm64 node.

- ingress-nginx: the cozystack controller and protobuf-exporter rebuild
goes to v1.11.6, and the vendored chart moves to the matching 4.11.6.
Upstream 1.11.6 is a maintenance release (Go, dependencies, NGINX base).
The template and Lua code the rebuild patches are unchanged since
1.11.5.
- cozy-proxy v0.4.0
- cozystack-scheduler v0.4.0
- keycloak-kms-proxy 0.2.3

The charts were re-vendored with `make update`. Every digest resolves to
an index that has both platforms.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(system): ingress-nginx v1.11.6, cozy-proxy v0.4.0, cozystack-scheduler v0.4.0 and keycloak-kms-proxy 0.2.3 now ship linux/arm64 images.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Updates**
  - Updated Cozy Proxy and Cozystack Scheduler to version 0.4.0.
- Updated Ingress-NGINX to 1.11.6, including its certificate-generation
component.
  - Updated Keycloak’s KMS proxy to version 0.2.3.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-29 00:32:18 +03:00
Aleksei Sviridkin 88b61c96de docs(kubevirt): state the preconditions of the eviction comments (#4441)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This fixes two comments in the KubeVirt CR that #4403 got wrong. It
follows up on the [review of
#4403](https://github.com/cozystack/cozystack/pull/4403#pullrequestreview-5301013108),
and I checked both findings against the source first. Only comments
change, the rendered CR is the same.

The comments didn't say that the new eviction behaviour applies only to
VMIs created after the switch. The VMI mutator copies the cluster
`evictionStrategy` into the VMI spec at creation (KubeVirt v1.9.0
`pkg/defaults/defaults.go:208-211`), and the eviction admitter reads the
spec first. So a non-migratable VMI created before the switch still has
`LiveMigrate` and keeps blocking the drain. The workload updater can't
evict it either, and it stays on the old launcher until its VM restarts.

The worker comment described the wrong mechanism. When a VMI can't
migrate and has `evacuationNodeName` set, CAPK's `IsTerminal()` returns
true and the controller sets `FailureReason` on the KubevirtMachine.
This is upstream CAPK code at the pinned commit, the cozystack patch
doesn't touch it. Every kubernetes-nodes group has a MachineHealthCheck,
and CAPI v1.10.1 treats `FailureReason` as unhealthy, so the Machine
gets replaced. If `maxUnhealthy` holds remediation back (at the default
`50%`, a group of one), the VM starts a new VMI in place. CAPK never
clears `FailureReason` though, so the Machine stays failed until
remediation is allowed.

The comment stays on the `evictionStrategy:` line, because `make update`
inserts blocks right before it. `helm unittest` passes 33/33 and
`update_idempotency_test.sh` passes. `make update` leaves the file
byte-identical.

The third finding (tests pin values, not behaviour) is tracked in #4422.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
docs(kubevirt): the KubeVirt CR comments now say that a VMI created before the evictionStrategy change keeps LiveMigrate until its VM restarts, and that the MachineHealthCheck replaces a non-migratable kubernetes-nodes worker when the workload updater evicts it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified that existing VM instances retain the eviction strategy they
were created with, and may need to be restarted before they can migrate
during a drain.
* Expanded guidance on how eviction of non-migratable workers affects
machine health and replacement, including when remediation is delayed by
the `maxUnhealthy` limit.
  * No rendered configuration values changed.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 23:09:22 +03:00
Aleksei Sviridkin d190dcec74 test(velero): check the KubeVirt plugin init container by repository
The release build stamps its own rebuild of the plugin over the
upstream ref, so the test matches the repository name alone. It still
fails when the init container is dropped or points at another image.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 21:56:58 +03:00
Aleksei Sviridkin 02461acdde Merge remote-tracking branch 'origin/main' into feat/arm64-bundle
Signed-off-by: Aleksei Sviridkin <f@lex.la>

# Conflicts:
#	hack/image-builders-cross-compile.bats
2026-09-28 21:42:18 +03:00
Aleksei Sviridkin 0040d6aa3d feat(keycloak): update keycloak-kms-proxy to 0.2.3 with arm64 images
0.2.3 publishes the proxy as a linux/amd64 and linux/arm64 index;
earlier releases were amd64 only.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 21:41:32 +03:00
Aleksei Sviridkin a0d3dba07f feat(cozystack-scheduler): update to v0.4.0 with arm64 images
v0.4.0 publishes the scheduler as a linux/amd64 and linux/arm64
index; earlier releases were amd64 only.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 21:41:32 +03:00
Aleksei Sviridkin c22cc2851e feat(cozy-proxy): update to v0.4.0 with arm64 images
v0.4.0 publishes cozy-proxy as a linux/amd64 and linux/arm64 index;
earlier releases were amd64 only.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 21:41:31 +03:00
Aleksei Sviridkin 75d7573942 feat(ingress-nginx): update controller to v1.11.6 with arm64 images
The cozystack controller and protobuf-exporter rebuilds are now
published for linux/amd64 and linux/arm64, so the ingress controller
can start on arm64 nodes. The rebuild is tagged by its upstream
version, and v1.11.6 is the first multi-arch one, so the chart moves
to the matching 4.11.6.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 21:41:31 +03:00
Aleksei Sviridkin b2ae6b773a fix(tenant): fail fast on an empty tenant apex and correct gateway docs (#4538)
Closes #4536

## What this PR does

Two fixes to how tenant charts handle the tenant apex host
(`_namespace.host`), from an audit of what happens when a tenant's
`host` is changed or left empty.

**1. Fail fast on an empty tenant apex (was: silent malformed
hostname).**
Charts that publish a service compose their hostname as
`<sub>.<_namespace.host>` (for example `harbor.<apex>`, `s3.<apex>`,
`grafana.<apex>`). When the injected apex was empty and no per-app
`host` override was set, they emitted a malformed trailing-dot hostname
(`harbor.`) into Ingresses, HTTPRoutes and Certificate dnsNames with no
error. Only the gateway path guarded against this.

A new library helper `cozy-lib.ns-host-required` now sources the apex at
each hostname-emitting site and fails the render with a clear message
when the apex is empty and no override is set. It is scoped to the
injected state, so standalone renders (`helm template` / `helm lint`
with no `_namespace`, or an empty `_namespace` map), no-exposure
tenants, and conditionally emitted resources (external=false, the
kubernetes no-etcd beacon path) are unaffected. The two post-delete
cleanup hooks are left unguarded so an empty apex cannot block
uninstall.

Charts covered: harbor, postgres (external), qdrant (external),
kubernetes (control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, grafana OIDC client), and system/bucket (which also
gains the `cozy-lib` library it was missing, both the local subchart
symlink and its package-source registration).

**2. Correct the tenant `gateway` parameter documentation.**
The `gateway` description (values.yaml, values.schema.json, README, the
Go type, the tenant-rd CRD, and an inline comment in namespace.yaml)
claimed the chart auto-enables a Gateway for tenants whose apex is
derived from the parent (empty `host`) and leaves it off for custom-apex
tenants. The resolver `tenant.gatewayEffective` never inspects `host`:
an unset `gateway` key always means the tenant owns no Gateway and
inherits from the nearest ancestor that owns one. The unit tests already
pin this behavior. The description is rewritten to match and
regenerated.

**Behavior change.** A tenant with an empty injected apex and no `host`
override that enables one of the covered components now fails its render
(the HelmRelease reports NotReady) instead of deploying a broken
hostname. In practice the apex is always injected (the root tenant has
`root-host`, children inherit it), so this only trips a genuinely
misconfigured tenant.

Tests: a new `cozy-lib.ns-host-required` unit test, plus per-chart guard
tests for harbor, qdrant, kubernetes, postgres, vpn, bootbox, seaweedfs,
monitoring and system/bucket. All affected chart suites pass.

### Screenshots

Not applicable (no UI change).

### Downstream repositories

I walked the trigger map against the diff. The only touchpoint is the
tenant `README.md` gateway description, which the website release bot
regenerates automatically from `README.md` on a stable tag (no manual
follow-up). The terraform provider is not affected: this is a
description-only change to the existing `gateway` field, with no field,
default or enum change, and the tenant-rd CRD schema reaches the
provider as opaque JSON. Boxes left for a maintainer to confirm.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tenant): tenant charts now fail rendering with a clear error when a tenant's apex host is empty and no per-app host override is set, instead of silently emitting a malformed hostname; the tenant `gateway` parameter description is corrected to match actual behavior (an unset `gateway` never auto-enables a Gateway).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Host-dependent services now show a clear rendering error when neither
a tenant host nor an app host is configured, instead of producing an
invalid hostname.
* Tenants without `gateway: true` use the nearest ancestor’s Gateway,
whether their host is inherited or custom. Enable `gateway: true` for a
tenant’s own Gateway, IP, and certificate; this is required to make a
custom apex publicly reachable.
* **Documentation**
* Clarified Gateway inheritance behavior and when a tenant needs its own
Gateway.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 21:32:36 +03:00
Aleksei Sviridkin f301cb551b feat(api): add configurable resources and replicas for cozystack-engine components (#2956)
## What this PR does

Exposes standard Helm `resources` values for the three core workloads in
the `cozystack-engine` package, and makes `cozystack-controller`
replicas configurable (was hardcoded to `1`).

**Changes by component:**

| Component | `resources` | `replicas` |
|---|---|---|
| `cozystack-api` | ✅ NEW (`cozystackAPI.resources`) | already
configurable |
| `cozystack-controller` | ✅ NEW (`cozystackController.resources`) | ✅
NEW (`cozystackController.replicas`, default `1`) |
| `lineage-controller-webhook` | ✅ NEW
(`lineageControllerWebhook.resources`) | already configurable |

Each template uses `{{- with .Values.xxx.resources }}` so the
`resources:` block is only rendered when values are provided. Default
`resources: {}` produces no output, so resource requests/limits stay
opt-in.

**Leader election:** `cozystack-controller` now always runs with leader
election — the chart passes `--leader-elect` unconditionally and ships a
namespaced `Role`/`RoleBinding` for the Lease in `cozy-system` (where
controller-runtime creates it). Raising `cozystackController.replicas`
therefore adds standbys instead of a second active writer over the
reconcilers that patch HelmReleases, Gateways, Certificates,
ResourceQuotas and Secrets; this holds whatever `replicas` is. A restart
can pause reconciliation for up to the Lease duration (15s by default),
because `LeaderElectionReleaseOnCancel` is off. With `replicas: 1` the
change for an existing install is the flag plus the two RBAC objects.

**Why:** Without these values, operators cannot set QoS guarantees and
must resort to `kubectl patch`, Flux Kustomize SSA patches, or Kyverno
mutations — all fragile under Helm upgrades.

Closes #2955

### Release note

```release-note
feat(api): add configurable `resources` (requests/limits) for cozystack-api, cozystack-controller, and lineage-controller-webhook.
`cozystack-controller` replicas is now configurable via `cozystackController.replicas` (default: 1).
The `cozystack-controller` Deployment now always runs with leader election and creates a Lease in `cozy-system`, so raising `replicas` adds standbys rather than a second active controller; a restart can pause reconciliation for up to the Lease duration (15s by default).
An empty `resources: {}` renders no resources block, so requests and limits remain opt-in.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Container resource requests and limits can now be configured for the
API, controller, and webhook services; defaults remain unchanged.
* The controller replica count is configurable, and leader election is
enabled for multi-replica operation.
* Controller leader-election access is limited to the `cozy-system`
namespace.

* **Documentation**
  * Documented the webhook resource configuration option.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 20:56:43 +03:00
Aleksei Sviridkin b75507bbf1 test(velero): stop asserting the KubeVirt plugin image string
The plugin image is now built in-tree and stamped by digest, so any
assertion on its string only restates the pin and breaks on every
build or bump without testing what the chart does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 20:04:11 +03:00
Aleksei Sviridkin 5a619c5fdc Merge remote-tracking branch 'origin/feat/velero-kubevirt-plugin-multiarch' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>

# Conflicts:
#	Makefile
2026-09-28 19:57:27 +03:00
Aleksei Sviridkin 8c1266649d Merge remote-tracking branch 'origin/feat/harbor-multiarch' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:55 +03:00
Aleksei Sviridkin 79854b16fb Merge remote-tracking branch 'origin/fix/build-list-orphan-images' into feat/arm64-image-builds
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 19:49:54 +03:00
Aleksei Sviridkin 6d759cc73e feat(build): build package images for amd64 and arm64 (#4498)
## What this PR does

Package images can now be built for amd64 and arm64 in one `make image`,
and a published build is multi-arch unless `PLATFORM` says otherwise.
This is the build half of arm64 support from #1961 and #903. CI still
publishes amd64 only until it gets a native arm64 leg, which comes in a
separate PR.

Builder stages now run on the build platform and compile for the target
one, so Go never runs under emulation. Under emulation it is slow and it
crashes: the migration-controller Dockerfile already records a "bad
pointer in Go heap" from exactly that. On top of this, `PLATFORM`
defaults to `linux/amd64,linux/arm64`. A build on an arm64 workstation
can't publish an arm64-only digest any more. That happened once already,
the cilium pin was arm64-only until 3f36a1b45 rebuilt it by hand.

Two images were only right because they were emulated. On a native
builder they would ship the wrong architecture. token-proxy ran `go
build` with no GOARCH, and kube-ovn calls upstream `make build-go`,
which hardcodes `GOARCH=amd64`. Both are fixed now. The new bats test
wants every compiling stage on `$BUILDPLATFORM` and actually using
`TARGETARCH`. A bare `ARG TARGETARCH` does not count, this is how
token-proxy got through.

Some things stay amd64. talos and testing pin it with `override`,
because matchbox serves amd64 Talos assets and the sandbox runs
`qemu-system-x86_64`. `LOAD=1` builds for the host only, since the
classic docker image store can't load an index. Every CI build job pins
`PLATFORM: linux/amd64`, so CI output does not change.

This changes local builds. A `make image` on the default docker driver
with the classic image store now fails with "Multi-platform build is not
supported for the docker driver", where before it built for the host.
Use a `docker-container` builder (`BUILDER=<name>`), or pass
`PLATFORM=linux/amd64` (or `LOAD=1`) to build one architecture.

I checked it on an arm64 host with a docker-container builder.
cozystack-api, dashboard, kubeovn, platform-migrations, cilium and
capi-providers-cpprovider build and push as amd64+arm64 indexes.
Binaries pulled from each platform are x86-64 and aarch64:
cozystack-api, token-proxy, the kamaji provider manager, cilium-agent
and all four kube-ovn binaries. piraeus-server builds for both
platforms, but the push to my test registry failed on registry 500s. The
capi-providers-cpprovider inline cache works with a multi-platform push.
Not covered is running the arm64 images on an arm64 cluster, CI has
none.

This is stacked on #4485 and targets its branch, so the diff here is
only the two commits on top.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#711
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(build): package images build for linux/amd64 and linux/arm64 by default, with Go builders cross-compiling on the build platform; a local `make image` now needs a docker-container builder, or PLATFORM=linux/amd64 / LOAD=1 for a single architecture. CI still publishes amd64.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Build Improvements**
* Container image builds now target both amd64 and arm64 by default when
building without loading images locally.
* CI and release builds are pinned to amd64; Talos and testing images
remain amd64-only.
* Cross-platform builds now compile on the builder’s native platform
while targeting the requested architecture.
* **Documentation**
* Clarified platform defaults and architecture requirements for local
and CI builds.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:43:47 +03:00
Aleksei Sviridkin 597d966d7d fix(redis): run the multi-arch redis_exporter so the sidecar starts on arm64 (#4522)
## What this PR does

Redis and Valkey instances, and the Redis inside Harbor, never become
fully ready on arm64 nodes. Their failovers pin
`oliver006/redis_exporter:v1.55.0-alpine`, which upstream publishes for
amd64 only. On arm64 the exporter sidecar exits with `exec
/redis_exporter: exec format error`, and the pod stays not ready.

The same release without the suffix is a multi-arch index with amd64,
arm and arm64 builds. Its entrypoint, user and port match the `-alpine`
image, and none of these charts run a shell in the exporter container.
So this swaps the tag and pins it by digest, with no version bump.

I checked both tags with `crane`. On an arm64 test cluster a Redis and a
Valkey instance hit the exec format error before the change and came up
2/2 with it. The Harbor Redis pods went to 2/2 the same way.

On upgrade the exporter image in the pod template changes, so the
operator rolls every Redis and Valkey instance and Harbor's Redis, one
pod at a time as it does for any failover update.

Fixes #4519

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes one image reference in three templates. No entry in the
trigger map covers a sidecar image.

### Release note

```release-note
fix(redis): Redis, Valkey and Harbor's Redis now use the multi-arch redis_exporter image, so the exporter sidecar starts on arm64 nodes.
```
2026-09-28 19:36:36 +03:00
Aleksei Sviridkin 17838d46dc fix(ingress): run the default backend on arm64 (#4517)
## What this PR does

The ingress default backend can't start on arm64 nodes: it crash-loops
with `exec /server: exec format error`. `packages/system/ingress-nginx`
turns the default backend on without setting an image, so it gets the
chart default `registry.k8s.io/defaultbackend-amd64:1.5`. That image is
amd64 only, and upstream never published a multi-arch defaultbackend.

This pins `registry.k8s.io/ingress-nginx/custom-error-pages:v1.2.9` by
its multi-arch index digest. Upstream ships it as the replacement
default backend, with amd64, arm and arm64 builds. It answers 404 on
:8080 and 200 on `/healthz`, so the chart's port and probes still fit.
The host ingress, the tenant ingresses and the tenant Kubernetes
ingress-nginx addon all render this package without their own image, so
this one change covers them.

I ran the arm64 image locally under the chart's securityContext (uid
65534, read-only root filesystem, all capabilities dropped). `/healthz`
returned 200, `/` returned 404, and memory stayed around 6Mi of the 20Mi
limit. The same override runs on an arm64 test cluster. A new
helm-unittest case checks the default backend image by digest; it fails
on main and passes here.

There are two side effects. custom-error-pages logs a line for every
request, because it looks for error page templates under `/www` that we
don't ship and falls back to a plain `404 page not found`. The old
backend didn't log, and its body was `default backend - 404`. The status
code is still 404.

Also, Renovate won't update this pin. The `helm-values` manager is
disabled in `.github/renovate.json`, and no regex manager matches the
registry/image/tag/digest layout. The controller image in the same file
has the same problem, so I kept Renovate config out of this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `packages/system/ingress-nginx/values.yaml` and a
new test in that package. No entry in the trigger map covers a system
package's image values.

### Release note

```release-note
fix(ingress): the ingress default backend now uses the multi-arch custom-error-pages image instead of the amd64-only defaultbackend image, so it starts on arm64 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Pinned the default error-page image to a specific version and digest,
improving consistency across deployments.
  * Added a chart test to verify the configured image.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:36:25 +03:00
Aleksei Sviridkin aa59c2f1d5 fix(foundationdb-operator)!: drop the amd64-only 7.1 init container (#4512)
## What this PR does

The FoundationDB operator never starts on arm64. Its
`foundationdb-kubernetes-init-7-1` init container runs
`foundationdb/fdb-kubernetes-monitor:7.1.67` and fails with `exec format
error`. All 67 published 7.1.x tags of that image are single amd64
manifests, so there is no multi-arch 7.1 tag to move to. The 7.3.63 and
7.4.1 tags, and the operator image itself, publish linux/arm64. Upstream
hits the same gap in its own build: the operator Dockerfile at v2.30.0
swaps its built-in 7.1 client for 7.3 on arm64.

The fix removes the 7.1 entry when the chart is vendored, instead of
overriding it in the package values. The `update` recipe deletes it with
`yq`, and the committed chart equals what the recipe produces. A `null`
override would not last. Helm 4.2 and later ignore a `null` that a
parent chart sets on a subchart key when no user-supplied values reach
that subchart (helm/helm#32132). The platform passes this package no
values, so that is exactly the case here. helm-unittest renders with
Helm 3, so the override would silently stop working after a Flux bump.
`helm template` now renders only the 7.3 and 7.4 init containers under
both Helm 4.1.1 (what helm-controller v1.5.0 uses) and Helm 4.3.0. The
test pins exactly two init containers, so it fails if a later `make
update` brings 7.1 back.

This is a breaking change. The operator loses the 7.1 `fdbcli`,
`fdbbackup`, `fdbrestore` and client library, so it can no longer manage
or upgrade a FoundationDB cluster pinned to 7.1.x. The app defaults to
7.3.63 and nothing in this repository sets 7.1, but `cluster.version` is
a free string. A user who set 7.1.x has to upgrade that cluster to 7.3
before installing this release.

Related: #4511, the app still accepts any string as the FoundationDB
version, 7.1 included.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches one system package: its update recipe, its vendored
chart values, and its test. No app schema, platform value, CRD or shared
tooling changes, and a search of the cozystack org finds no page or code
that offers FoundationDB 7.1.

### Release note

```release-note
fix(foundationdb-operator)!: the FoundationDB operator now starts on arm64. It no longer ships the 7.1 client, because no arm64 build of it exists, so it cannot manage or upgrade a FoundationDB cluster on 7.1.x. Upgrade such clusters to 7.3 before installing this release.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* The FoundationDB operator chart no longer includes the 7.1
init-container configuration. Its remaining configured init-container
versions are 7.3 and 7.4.
* The chart update process applies the same version change to copied
chart values.
* Image validation now checks for exactly two init containers, using
versions 7.3.63 and 7.4.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:36:19 +03:00
Aleksei Sviridkin 2e8219fb92 fix(build): pass the shared buildx flags from every image target (#4485)
## What this PR does

Five image builds skipped the shared buildx flags in
`hack/common-envs.mk` and wrote their own: flux-plunger,
kubeovn-plunger, both dashboard images, and metallb. The plungers
ignored `PLATFORM`, so on an arm64 workstation they came out as arm64
images that fail on amd64 nodes with `exec format error`. CI builds on
amd64, so it never showed. Dashboard and metallb hardcoded
`--platform=linux/amd64`; for metallb that sat next to `$(BUILDX_ARGS)`,
so setting `PLATFORM` gave buildx two platforms. All of them now build
through `$(BUILDX_ARGS)` only.

How it split: kubeovn-plunger landed two days before 9f2b98d36 moved the
other Makefiles onto `BUILDX_ARGS`, flux-plunger was copied from it, and
dashboard kept its own flags when that change touched it.

CI builds do not change. CI sets only `BUILDER`, which these targets
already honoured, and every build runner is x86-64.

One behaviour change is intentional: a local `make image` for dashboard
or metallb now follows the host architecture, like every other package.
On an arm64 machine, build for an amd64 cluster with
`PLATFORM=linux/amd64`.

A new bats test dry-runs every package image target with a probe
platform and a marker in `BUILDX_EXTRA_ARGS`. Each buildx call must
carry the marker and exactly that platform. On main it flags the five
targets above.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(build): every package image build honours PLATFORM, SBOM and BUILDX_EXTRA_ARGS; local dashboard and metallb builds now follow the host architecture unless PLATFORM is set.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* System image builds now use shared Buildx settings for platform
selection and publishing options, while retaining their image tags,
cache settings, and metadata.
* MetalLB image builds no longer explicitly target `linux/amd64`; the
selected platform follows Buildx defaults and configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 19:36:07 +03:00
Aleksei Sviridkin 9c94337f17 fix(mariadb): correct webhook cert-manager override key path (#4537)
## What this PR does

Fixes cozystack/cozystack#3974: the values override at
`packages/system/mariadb-operator/values.yaml` targeted
`webhook.certificate.certManager`, a field that does not exist in the
vendored mariadb-operator chart. The chart's actual key is
`webhook.cert.certManager.enabled` (see
`charts/mariadb-operator/values.yaml:139-142`), so the override was
silently ignored and the webhook always fell back to mariadb-operator's
own cert-controller instead of cert-manager. This corrects the key path
so the override takes effect.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(mariadb): correct webhook cert-manager override key path so cert-manager is actually used for the mariadb-operator webhook certificate
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the MariaDB operator’s certificate-management configuration to
use its current setting structure. The certificate manager remains
enabled, now under the updated configuration path. Installations that
use the operator will apply the enabled setting from the revised values
configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 18:53:51 +03:00
Aleksei Sviridkin 26e4083d52 fix(linstor): adopt existing CR on 409 AlreadyExists in k8s backend (#4532)
## What this PR does

This adds a LINSTOR server patch for a controller that could refuse
every write of one property until it was restarted, which left a PV
stuck in `Released` with its ZFS volume still allocated. It is a
backport of LINBIT/linstor-server#531.

linstor-csi starts every volume delete by setting
`BalanceResourcesEnabled` on the resource definition. With the
Kubernetes CRD backend that write can come back as 409 AlreadyExists
when the fabric8 client retries a POST the apiserver already persisted.
A single-object transaction has no rollback entry, so the controller
drops the key from memory while the CR stays, and the caching client
never re-reads it. Every retry of the delete then fails the same way.

The patch makes create treat a 409 as success when the object exists: it
takes the live resourceVersion and replaces the object with the intended
content. The unit test that comes with the patch was red before the fix
and green after, on v1.33.3 and on upstream master, and the whole patch
series still applies to v1.33.3.

The retry being the trigger is my best explanation, not proven: the run
had no apiserver logs. Until this ships, restarting linstor-controller
clears the state.

Closes #4531

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): the LINSTOR controller no longer gets stuck refusing writes to a property after a retried Kubernetes API request, which could leave a deleted volume in Released with its storage still allocated.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved recovery when Kubernetes reports that an object already
exists during creation. The system now adopts the existing object and
continues updating it, helping prevent volume operations from becoming
stuck and requiring a controller restart.
* Other conflict and error cases continue to report their original
errors.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 18:39:43 +03:00
IvanHunters 2fa51bf587 test(charts): isolate guard coverage for every guarded template
Add per-template guard suites so each guarded hostname site is exercised
on its own: harbor harbor.yaml and httproute.yaml, bootbox machines.yaml,
seaweedfs cosi-deployment.yaml, bucket deployment.yaml and httproute.yaml,
monitoring alerta.yaml, grafana httproute.yaml and the grafanaHost helper
via oidc-keycloak.yaml. Each suite renders only its target template, so
removing that file's guard reddens it. Also add a postgres positive case
that renders the Certificate with an apex set (external=true).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 18:32:49 +03:00
Aleksei Sviridkin fa6e3ce850 docs(kubernetes): clarify node pool Helm timeout semantics (#4496)
## What this PR does

Corrects the explanation for the Kubernetes node-pool Helm timeout.

With readiness waiting disabled, nothing in this chart's install or
upgrade is bounded by the 15-minute timeout: the ordinary
`talos-reconcile` Job continues asynchronously, and the chart has no
install or upgrade hooks. The timeout takes effect only if readiness
waiting is re-enabled or such a hook is added.

The annotations and their values are unchanged.

Fixes #4406

### Screenshots

N/A — no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified that with readiness waiting disabled, the 15-minute Helm
timeout does not limit chart installs or upgrades: reconciliation runs
asynchronously, and the chart has no install or upgrade hooks.
* Noted that the timeout applies if readiness waiting is re-enabled or a
hook is added, and that the release settles while reconciliation
continues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 17:55:57 +03:00
Aleksei Sviridkin ec9843be4a docs(virtual-machine): correct persistent EFI migration documentation (#4494)
## What this PR does

Removes the outdated warning that persistent EFI state on default RWO
storage makes a virtual machine node-pinned and prevents live migration.
The warning no longer reflects the supported migration behavior and
could discourage users from persisting Secure Boot keys.

This is a documentation-only change. The chart documentation, JSON
schema, generated API comment, and ApplicationDefinition schema were
regenerated without changing defaults, validation, or runtime behavior.

Fixes #4420

### Validation

- `make -C packages/apps/vm-instance generate`
- `make generate`
- `make manifests`
- `make -C packages/apps/vm-instance test` — 59 tests passed
- `git diff --check`
- A second complete generation pass produced no additional changes

### Screenshots

N/A — documentation-only change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Updated EFI persistence descriptions to remove claims that using
default RWO storage pins virtual machines to a node or prevents live
migration. The persistence behavior and UEFI-only applicability remain
unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-28 17:53:50 +03:00
Nikita Aboltin cb8728dde1 docs: note the measured idle usage next to the empty resources defaults
The resources keys stay empty so requests and limits are opt-in, and the comment
records the measured idle usage next to them, so a limit that does get set is not
placed below what the component needs at idle.

Signed-off-by: Nikita Aboltin <aboltin.nikita@rwb.ru>
2026-09-28 16:43:44 +03:00
IvanHunters 0f4d39f1ef test(charts): add empty-apex guard tests for the remaining guarded charts
Add host_guard_test.yaml for postgres, vpn, bootbox, seaweedfs,
monitoring and bucket. Each renders its guarded template with a
non-empty injected _namespace whose host is empty and no per-app host
override, and asserts the ns-host-required failure, plus a positive
render case with the apex set. Completes guard coverage alongside the
existing harbor, qdrant and kubernetes tests.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 16:01:31 +03:00
IvanHunters eedb446e8a docs(tenant): clarify that gateway=false behaves like unset
The rewritten gateway description claimed setting false opts out of
Gateway publishing entirely, including inheritance. It does not:
tenant.gatewayEffective returns "false" for both an explicit false and
an unset key, so namespace.yaml keeps _namespace.gateway pointing at the
nearest ancestor that owns a Gateway and the tenant still publishes
through it. State that false currently behaves the same as unset, and
fix the matching stale comment in gateway_default_test.yaml.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:52:46 +03:00
Nikita Aboltin 8732dc96cf feat(api): add configurable resources and replicas for cozystack-engine components
- cozystack-api, cozystack-controller, lineage-controller-webhook: conditional
  resources block via `{{- with .Values.*.resources }}`, defaulting to `{}` so the
  default render stays unchanged
- cozystack-controller: replicas comes from values (was hardcoded 1); the chart
  now passes --leader-elect unconditionally and grants the Lease verbs through a
  namespaced Role and RoleBinding in cozy-system (the Lease lives in the manager's
  own namespace), so raising replicas adds a standby instead of a second active
  writer over the reconcilers that patch HelmReleases, Gateways, Certificates,
  ResourceQuotas and Secrets

Signed-off-by: Nikita Aboltin <aboltin.nikita@rwb.ru>
2026-09-28 15:19:45 +03:00
IvanHunters 37f59369d3 fix(charts): fail fast on empty tenant apex in host composition
Charts that publish a service compose their hostname as
<sub>.<_namespace.host>. When the injected tenant apex is empty and no
per-app host override is set, they used to emit a malformed trailing-dot
hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate
dnsNames silently. Source the apex through cozy-lib.ns-host-required at
each hostname-emitting site so the render fails with a clear message
instead, matching the gateway path.

Covered: harbor, postgres (external), qdrant (external), kubernetes
(control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, oidc client), and system/bucket (adds the cozy-lib
subchart symlink it lacked). The guard is placed at the point each
resource is actually emitted, so standalone renders, no-exposure
tenants and the kubernetes no-etcd beacon path are unaffected. The
post-delete cleanup hooks are left unguarded so an empty apex cannot
block uninstall.

Behavior change: a tenant with an empty injected apex and no host
override that enables one of these components now fails its render
(HelmRelease NotReady) instead of deploying a broken hostname. In
practice the apex is always injected (root has root-host, children
inherit it), so this only trips a genuinely misconfigured tenant.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 15:14:45 +03:00
chathurabuddi 44a06486a9 fix(mariadb): correct webhook cert-manager override key path
The chart schema is webhook.cert.certManager.enabled, not
webhook.certificate.certManager, so the override was silently
ignored and the webhook fell back to mariadb-operator's own
cert-controller instead of cert-manager.

Assisted-by: LLM
Signed-off-by: chathurabuddi <10394110+chathurabuddi@users.noreply.github.com>
2026-09-28 17:41:20 +05:30
IvanHunters 784d290c42 docs(tenant): correct gateway param description to match behavior
The gateway parameter description claimed the chart auto-enables a
Gateway for tenants whose apex is derived from the parent (empty host)
and leaves it off for custom-apex tenants. The resolver
tenant.gatewayEffective never inspects host: an unset gateway key
always means the tenant owns no Gateway and publishes through the
nearest ancestor that owns one, regardless of host. The unit tests in
gateway_default_test.yaml already pin this real behavior.

Rewrite the description to match the code and regenerate the schema,
README, Go types and the tenant-rd CRD.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-28 14:07:51 +03:00
Aleksei Sviridkin 11991b9d05 feat(harbor): build the Harbor images for amd64 and arm64
Upstream publishes the Harbor v2.15 component images for amd64 only,
so every Harbor pod crash-loops with "exec format error" on an arm64
node. The multi-arch build upstream exists on main but is in no
release yet, so the package now rebuilds the pinned v2.15.1 release
itself: the Go components cross-compile on the build platform, the
registry and the Trivy adapter build from the tags upstream's Makefile
pins, the Trivy binary is fetched per architecture against its release
checksum, and the runtime images keep upstream's photon base, users
and entrypoints. The build fails when those registry and Trivy pins,
the swagger generator or the golang and node build images no longer
match upstream's Makefile at the pinned release, so a version bump
cannot pair a new core with an old registry, scanner or toolchain.

The portal compiles to static JS, so it lives in its own Dockerfile
listed as architecture-neutral in the cross-compile check, and takes
the release pin from the main one so both files build the same
commit.

Because values.yaml now pins the images, `make update` pins the chart
release that ships the same Harbor version and fails when its
appVersion differs, instead of pulling the newest chart onto old
images. Renovate stops proposing tag bumps for the rebuild's build
bases, which follow upstream's toolchain for that release.

The committed refs stay on the upstream digests until CI stamps the
rebuilt ones.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 02:11:46 +03:00
Aleksei Sviridkin f02963a2c7 fix(keycloak-operator): keep the image host inside repository
The image target stamped only `tag`, leaving the committed
`registry: ghcr.io` in place, so a build against the CI registry would
render the public host with a digest that exists only in the build
registry. Stamping the build host into `registry` instead would split
the source registry across two keys, where the nightly mirror's
substring host rewrite cannot reach it and the published tree would
keep pointing at the private registry.

Set `registry: ""` and keep the whole host in `repository`, stamped
by the image target. The upstream chart joins the two keys, so the
rendered image string is unchanged, and the ref now has the same
shape as every other stamped reference the mirror and promotion
already handle.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 00:57:24 +03:00
Aleksei Sviridkin 11ba78a2b9 fix(linstor): adopt existing CR on 409 AlreadyExists in k8s backend
The fabric8 client retries a POST after a 5xx or a dropped
connection, so a create that the apiserver already persisted can come
back as 409 AlreadyExists. A single-object transaction has no
rollback entry, so the controller drops the key from memory while the
CR stays in the apiserver, and the caching client never re-reads it:
every later write of that key fails the same way until the controller
restarts.

This left a CDI scratch PV stuck in Released, because linstor-csi
starts every volume delete by setting BalanceResourcesEnabled on the
resource definition and that write never succeeded.

Treat 409 on create as success when the object exists: take its
resourceVersion and replace it with the intended content.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 17:50:59 +03:00
frolov.so 15fee54558 docs(kubernetes): clarify node pool Helm timeout semantics
Readiness waiting is disabled for node-pool installs and upgrades, so
the ordinary reconciliation Job runs asynchronously and the configured
timeout is inert. Saying otherwise obscures the actual failure boundary.

Fixes cozystack/cozystack#4406

Assisted-by: LLM
Signed-off-by: frolov.so <opsmon.sh@yandex.com>
2026-09-27 17:18:01 +03:00
Aleksei Sviridkin dae5d5c5ae fix(build): name the repository of the keda and kuberture pins
Both packages pinned their images by writing tag@digest into the tag
field alone and left the repository to the vendored chart's defaults.
Tooling that reads refs from the package values saw a bare
"2.20.2@sha256:..." with no image to resolve, so the release gate
could not tell whether these images serve arm64. The values now
repeat the chart's registry and repository beside the pin; the
rendered manifests are unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 07:57:31 +03:00
Aleksei Sviridkin 769d3b3897 fix(build): push the kamaji provider image under the build's IMAGE_TAG
The capi-providers-cpprovider Makefile assigned IMAGE_TAG itself, which
beats the value CI puts in the environment. Every PR, main and release
line build therefore pushed the same component-version tag and
overwrote each other's, and its second tag was built from an undefined
TAG variable and ended in a bare "-".

The image now goes through image-tags like every other package: the
build-unique IMAGE_TAG always, the component version only under
PUBLISH_VERSIONED=1. The stamped refs keep the component version as
their tag, so the promote, retag and mirror tooling see the same refs
as before.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 07:57:31 +03:00
Aleksei Sviridkin 07700c74ce fix(redis): run the multi-arch redis_exporter so the sidecar starts on arm64
The redis, valkey and harbor Redis failovers pin
oliver006/redis_exporter:v1.55.0-alpine, which upstream publishes for
amd64 only. On an arm64 node the exporter sidecar exits with an exec
format error, so the Redis pod never becomes fully ready.

The unsuffixed v1.55.0 tag is a multi-arch index (amd64, arm, arm64)
of the same exporter release, with the same entrypoint, user and
port, and none of these charts run a shell in that container. Pin it
by digest, so the reference is reproducible.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-27 03:04:44 +03:00
Aleksei Sviridkin d4dfae656f fix(ingress): run the default backend on arm64
The ingress-nginx chart defaults the default backend to
registry.k8s.io/defaultbackend-amd64, which ships no arm64 build and
crash-loops there with "exec format error". Every ingress controller
rendered from this package, the host and tenant ingresses and the
tenant Kubernetes addon, enables that backend without naming an image.

Pin the upstream multi-arch replacement, custom-error-pages, by its
index digest. It answers 404 on :8080 and serves /healthz, so the
chart's port, probes and restricted securityContext apply unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 22:32:06 +03:00
Aleksei Sviridkin 9eb3a64dd1 fix(foundationdb-operator)!: drop the amd64-only 7.1 init container
Every foundationdb/fdb-kubernetes-monitor 7.1.x tag is a single amd64
image, so on arm64 the operator's 7.1 init container fails with exec
format error and the operator pod never starts. No multi-arch 7.1
monitor exists to bump to, and upstream already swaps its own 7.1
client for 7.3 when building the operator for arm64.

The vendoring recipe now deletes the 7.1 entry from the upstream chart
values, and the committed chart matches its output. A null override in
the package values was rejected: Helm 4.2 and later ignore a null the
parent chart sets for a subchart key when no user-supplied values reach
that subchart (helm/helm#32132), and the platform passes this package
none. helm-unittest renders with Helm 3, so the override would stop
working on a Flux bump without any test noticing.

The app defaults to 7.3.63, which keeps its init container, as does
7.4.

BREAKING CHANGE: the operator no longer carries the 7.1 fdbcli and
client library, so it can neither manage nor upgrade a FoundationDB
cluster pinned to a 7.1.x version. Upgrade such clusters to 7.3 before
installing this release.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 20:30:50 +03:00