Commit Graph
214 Commits
Author SHA1 Message Date
Andrey Kolkov 07301c562c fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names (#5083)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #5014.

A bucket's `Workload` was named `"bucket-" + <BucketClaim name>`, and a
Bucket application names its claim after its release, which already
starts with `bucket-`. A bucket an operator knows as `cozy-backups`
therefore appeared as `bucket-bucket-cozy-backups`, and that string is
the `workload` of the bucket's consumer in every billing usage report.

- **Naming.** The `bucket-` prefix is now added only when the claim name
does not already carry it. A Bucket application's claim
`bucket-cozy-backups` gives the `Workload` `bucket-cozy-backups`; a
claim without the prefix, such as `my-bucket`, still gives
`bucket-my-bucket`. In the tree today every monitored BucketClaim comes
from the Bucket application (Harbor's registry claim carries no labels
its monitors select, so it has no `Workload` on either side of this
change).
- **Migration of existing Workloads.** Nothing garbage-collects bucket
`Workload`s by name: they are owned by their BucketClaim and live as
long as it does. Without a migration the old `bucket-bucket-…` object
would stay next to the new one and the bucket would be metered twice.
When the new `Workload` is reconciled, the controller looks up the
legacy name and, only if the same BucketClaim (matched by UID) is its
sole owner, uses its recorded sizes as the starting point and then
deletes it, with UID and resourceVersion preconditions so a `Workload`
that took over the name in the meantime survives. If the legacy
`Workload` has other owners as well, it is another bucket's renamed
`Workload` that a rolled-back controller attached this claim to: the
controller takes neither its sizes nor the object and only drops its own
owner reference. A legacy `Workload` with no owners left (a claim
deleted with `--cascade=orphan` and recreated) that still carries this
monitor's label is deleted the same way, without taking its sizes. Sizes
already recorded at the new name win over the legacy ones only when the
claim already owned that `Workload`. Carrying the sizes over matters
because the controller deliberately keeps the last known size when the
object-store metrics are unavailable; a fresh `Workload` would otherwise
report no size until the next successful poll, which billing reads as an
empty bucket.
- **Never taking over another claim's Workload.** With Bucket
applications `x` and `bucket-x` in one namespace, the legacy name of the
first (`bucket-bucket-x`) is the new name of the second. The controller
never takes over a `Workload` that has owners when the claim is not one
of them (in practice another BucketClaim). While the new name is held
that way, it keeps metering the bucket under its legacy name, without
migrating, and emits a `BucketWorkloadNameTaken` warning on the monitor.
For Bucket applications the other claim renames its `Workload` away on
its own reconcile and the migration completes on the next pass. Only
when the legacy name is held as well — which for a claim without the
prefix, whose two names coincide, means a claim `x` outside the Bucket
application next to the application `x` — is the bucket skipped, and the
warning says it is not metered. Without this, whichever claim reconciled
first decided whether the second bucket ended up with the first one's
size or with no size at all. The same rule keeps two claims that map to
one name (`x` and `bucket-x`) from sharing a `Workload`.

**Rollback.** The migration is one-way. A controller rolled back to a
previous version recreates `bucket-bucket-<name>` next to the renamed
`bucket-<name>`, which its monitor keeps alive, so the bucket is metered
twice until the `bucket-<name>` Workloads of Bucket applications are
deleted by hand. Upgrading again after such a rollback converges without
deleting another bucket's `Workload` (see above).

**Known limitation.** Taking over an ownerless `Workload` keeps the
sizes recorded on it as the last known ones, as the controller always
has. If such an object was left behind by another bucket (for example
`kubectl delete bucketclaim --cascade=orphan`), its sizes count for the
new claim until the next successful size poll.

**Effect on billing, worth knowing before the upgrade:** `Workload`
names are the identity billing integrates over. A bucket's consumer
changes name once, at the upgrade, so a usage report whose window spans
the upgrade lists the bucket twice: under the old name up to the switch
and under the new one after it. The totals are right; the split is
cosmetic. Reports for windows entirely before or after the upgrade are
unaffected.

Tests (`internal/controller/workloadmonitor_controller_test.go`):

- `TestBucketWorkloadName`: the naming rule for a Bucket application
claim, a plain claim and a Harbor claim.
- `TestReconcileBucketApp_NoDoublePrefix`: a Bucket application gets
`bucket-cozy-backups` and no `bucket-bucket-cozy-backups`.
- `TestReconcileBucketApp_ReplacesLegacyWorkload`: an existing
`bucket-bucket-cozy-backups` owned by the claim is deleted, and its
recorded logical and physical sizes (no metrics endpoint in the test)
are carried over to the new `Workload`.
- `TestReconcileBucketApp_KeepsLegacyNameOwnedByAnotherClaim`: a
`Workload` under the legacy name owned by a different claim is left
alone and its size is not taken.
- `TestReconcileBucketApps_LegacyNameIsAnotherAppsNewName`: Bucket
applications `x` and `bucket-x` with their legacy `Workload`s,
reconciled in both orders, end with `bucket-x` and `bucket-bucket-x`,
each with one owner and its own size, and the legacy
`bucket-bucket-bucket-x` gone.
- `TestReconcileBucketClaim_DoesNotAdoptAnotherClaimsWorkload`: a claim
whose name is held by another claim's `Workload` leaves it untouched,
emits `BucketWorkloadNameTaken` and is metered under its legacy name.
-
`TestReconcileBucketClaim_KeepsMeteringUnderLegacyNameWhileNewNameIsHeld`:
the legacy `Workload` keeps being updated, with its size and owner,
across repeated reconciles while the new name is held.
- `TestReconcileBucketClaim_UnprefixedClaimWithHeldNameIsSkipped` and
`TestReconcileBucketClaim_BothNamesHeldIsSkipped`: when no free name is
left, nothing is taken over and the event says the bucket is not
metered.
- `TestReconcileBucketClaim_UnprefixedClaimKeepsItsWorkload`: for a
claim without the prefix, where the legacy and the new name coincide,
its own `Workload` and size survive repeated reconciles.
- `TestReconcileBucketApp_RecordedSizeWinsOverLegacy`: a size already
recorded on the new `Workload` takes precedence over the legacy one.
- `TestReconcileBucketClaim_UpdatesItsOwnWorkload`: a claim's own
`Workload`, with its owner reference carrying the `BucketClaim` kind as
a live cluster's cache records it, keeps being updated and raises no
warning (the fake client leaves that kind empty on objects the
controller creates, so the other tests do not exercise this).
- `TestReconcileBucketClaim_AdoptsOwnerlessWorkload`: a `Workload` that
lost all owner references is taken over by the claim whose name it
carries, rather than blocking that name forever.
- `TestReconcileBucketApp_ReleasesSharedLegacyWorkload`: a legacy
`Workload` that another claim also owns (the rollback case) keeps its
size and its other owner; only this claim's reference is dropped, and
its size is not carried over.
- `TestReconcileBucketApp_KeepsWorkloadThatReplacedLegacy`: a `Workload`
that replaces the legacy one under the same name between the lookup and
the delete survives.
- `TestReconcileBucketApp_RemovesOrphanedLegacyWorkload`,
`TestReconcileBucketApp_KeepsOrphanOfAnotherMonitor`: an ownerless
legacy `Workload` with this monitor's label is removed without carrying
its sizes; one labelled for another monitor is left alone.
- `TestReconcileBucketApp_LegacySizeWinsOverOwnerlessNewWorkload`: a
stale size on an ownerless `Workload` at the new name loses to the
migrating legacy `Workload`'s size.
- `TestReconcileBucketApp_LegacyAlreadyGone`: a legacy `Workload` that
disappears between the lookup and the delete (NotFound) does not fail
the per-claim reconcile, and the size is still carried over.

Reverting the naming rule, the legacy delete, the size carry-over, the
owner check, the refusal to adopt, the same-name guard, the size
precedence ignoring NotFound on the legacy delete, the own-claim match
in the adoption check, the ownerless case, the physical size carry-over,
the sole-owner rule for legacy Workloads, the delete preconditions, the
legacy-name fallback or its own held check, the ownerless-legacy rule or
the size precedence on takeover each fails one of these. `go test
./internal/... -count=1` and `go vet ./internal/controller/` pass.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, which touches only
`internal/controller/workloadmonitor_controller.go` and its test: no
package, values, schema, CRD, variant, `hack/` tooling, telemetry metric
or cozy-proxy contract changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(workloadmonitor): Workloads of Bucket applications are no longer named with a doubled "bucket-" prefix (bucket-bucket-<name> becomes bucket-<name>). Existing Workloads are replaced on upgrade, keeping their recorded size; billing usage reports whose window spans the upgrade list such a bucket once under each name. Rolling the controller back afterwards requires deleting the bucket-<name> Workloads of Bucket applications, or the buckets are metered twice.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Bucket workloads now use a consistent name without duplicating the
`bucket-` prefix.
* Existing workload size data is retained when workloads are renamed or
updated.
* Workloads owned by another claim are preserved. When a naming conflict
occurs, metering continues under the legacy name when possible; if both
names are unavailable, metering is skipped and a warning is emitted.
* Legacy workloads are cleaned up or updated according to their
ownership, helping prevent duplicate or orphaned workloads.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 13:43:42 +04:00
Myasnikov Daniil 8d8179f20a fix(controller): carry healthCheckExprs and waitStrategy to existing releases
cozystack-api writes healthCheckExprs and the resolved wait strategy
from the ApplicationDefinition only when it builds a HelmRelease, and
the definition reconciler that keeps existing releases in step copied
neither. A release created before its definition exempted an object
kept waiting on that object at every upgrade. For vm-disk that is
every disk created before the DataVolume expression was added: a disk
whose import is still running after a platform upgrade fails the
upgrade at the timeout and, under RetryOnFailure, retries forever
while holding a helm-controller worker.

The reconciler now derives both fields the way the API does, in both
directions, so an existing release ends up with what a fresh write
would produce. On the shipped definitions this only relaxes waits: the
vm-disk DataVolume and the postgres KEDA ScaledObject. A release that
another chart renders carries Flux's origin label and keeps the gate
its chart sets.

A failed update used to be logged and dropped, and nothing else
re-enqueues the definition for hours, so that release stayed out of
sync. The errors are now returned and the definition is retried.

Fixes https://github.com/cozystack/cozystack/issues/4409

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 14:35:34 +00:00
Andrey Kolkov a6c14f6834 fix(workloadmonitor): migrate ownerless legacy bucket Workloads and prefer legacy sizes on takeover
A BucketClaim deleted with --cascade=orphan and recreated leaves its
bucket-bucket-<name> Workload without owners. The migration recognised a
legacy Workload only by the claim's UID among its owners, so that object
stayed next to the new one with a frozen size and the bucket was metered
twice. Treat an ownerless legacy Workload that carries the monitor's
label as the claim's: delete it with the same preconditions, but take no
sizes from it, since they may predate the claim.

The size carry-over kept whatever the Workload at the new name had
recorded. When that Workload was not yet owned by the claim, an ownerless
leftover, its sizes were not written for this bucket, yet they won over
the migrating legacy Workload's. Prefer the recorded sizes only when the
claim already owned the Workload.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 17:45:04 +04:00
Andrey Kolkov 36ce21db79 fix(workloadmonitor): keep metering a bucket under its legacy name while the new one is taken
When the new Workload name was held by another owner the claim was
skipped altogether, so its legacy Workload stopped being refreshed and
billing kept reading the last size it had recorded. That happens briefly
with Bucket applications "x" and "bucket-x" until the first one has
migrated, and for good with a claim "x" outside the Bucket application
next to the application "x", which map to the same name.

Keep reconciling the bucket under its legacy name instead, without
migrating, until the new name is free. Skip it only when the legacy name
is held by another owner as well, which is the only option left for a
claim without the prefix, whose two names coincide; the warning event
then says that the bucket is not metered.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 17:34:25 +04:00
Andrey Kolkov fd69a8fbd6 fix(workloadmonitor): release, not delete, a legacy bucket Workload shared with another claim
A controller rolled back to the doubled prefix runs CreateOrUpdate on
bucket-bucket-x, which by then is the renamed Workload of the Bucket
application "bucket-x", and appends claim bucket-x as a second owner.
After the next upgrade claim bucket-x took that object for its own
legacy Workload: it carried the other bucket's size over and deleted the
object, and the other bucket's Workload came back without a size.

Treat a legacy Workload as the claim's own only when the claim is its
sole owner. When it has other owners, take neither its sizes nor the
object: drop the claim's owner reference and leave the rest.

Delete the legacy Workload with UID and resourceVersion preconditions,
so a Workload that took over the name between the lookup and the delete
survives; a conflict is not an error and the next reconcile retries.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 17:06:57 +04:00
Andrey Kolkov f9b7e936d7 refactor(workloadmonitor): leave alone any bucket Workload the claim does not own
The adoption check only refused a Workload owned by another BucketClaim,
and the kind test in it had no case that could tell it apart: a bucket
Workload only ever has BucketClaim owners. Refuse any Workload that has
owners when the claim is not among them, which needs no kind at all, and
keep taking over one that has lost every owner reference, since nothing
else would ever release its name. Pin the ownerless case.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 16:55:32 +04:00
Andrey Kolkov 54a916e31f test(workloadmonitor): pin that a claim keeps updating its own bucket Workload
On a live cluster the cache records the BucketClaim kind in a Workload's
owner reference, so a claim's own Workload passes through the "owned by
another claim" check on every reconcile, and only the own-UID match keeps
it from being skipped. The fake client leaves that kind empty, so no test
exercised it. Seed a Workload whose owner reference carries the kind and
the claim's UID, and expect it to be updated with no warning event.

Pin the physical size carry-over alongside the logical one, and word the
warning event so it holds when two claims map to one name for good.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 16:50:39 +04:00
Andrey Kolkov 730135903f test(workloadmonitor): pin that a vanished legacy bucket Workload is not an error
The legacy Workload can disappear between the lookup and the delete, for
example through owner garbage collection, and the delete ignores
NotFound for that reason. Reconcile only logs per-claim errors, so check
reconcileBucketClaimForMonitor's own result with a delete that reports
NotFound.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 15:43:29 +04:00
Andrey Kolkov a51ad0ed54 fix(workloadmonitor): never adopt a bucket Workload owned by another claim
With Bucket applications "x" and "bucket-x" in one namespace, the legacy
name of the first, bucket-bucket-x, is the new name of the second. When
the second reconciled first, CreateOrUpdate took over the first one's
legacy Workload, which already carried bucket x's size, and appended a
second owner. The first claim then found its own UID among the owners,
treated the object as its legacy Workload and deleted it, and the
second bucket's Workload came back without a size.

Leave a Workload alone while it is owned by a different BucketClaim, log
it and emit a BucketWorkloadNameTaken warning on the monitor. In the
case above the other claim renames its Workload away on its own
reconcile, and the name is free on the next pass. The same refusal keeps
two claims that map to one name, such as "x" and "bucket-x", from
sharing one Workload.

Pin the edge cases the migration depends on: the reconcile order above
in both directions, a claim without the prefix keeping its own Workload,
and a recorded size taking precedence over the legacy one.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 15:06:36 +04:00
Andrey Kolkov 164b5ec859 fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names
A bucket's Workload was named "bucket-" + the BucketClaim name, and a
Bucket application names its claim after its release, which already
starts with "bucket-". A bucket an operator knows as cozy-backups
showed up as bucket-bucket-cozy-backups in every billing usage report.

Add the prefix only when the claim name does not carry it. Claims from
other charts, such as Harbor's <release>-registry, keep their names.

Existing Workloads under the doubled name are owned by their claim and
would otherwise live on next to the new one, metering the bucket twice.
When the replacement is reconciled, take the legacy Workload's recorded
sizes as the starting point, so the rename leaves no gap in the metered
size, and delete the legacy Workload. Only a Workload owned by the same
BucketClaim is touched.

Billing sees the renamed Workload as a new consumer: a usage report
spanning the upgrade lists the bucket once under each name.

Fixes #5014

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 14:50:10 +04:00
Alexey Artamonov 699555c7cb fix(fluxcd): give each release its own share of the hook's retries
The pre-delete hook's retry budget was one window spent in listing
order, so a release that could never be read or unfenced, listed
first, took all of it and every release after it got a single attempt.
The same transient failure then released a fence or not depending on
where its release sorted. Each release now gets its own share of the
window, capped at the window's end, and the steps that cover every
release keep the whole of it.

Two comments said more than holds. The retry window is measured from
the script's start, not the Job's, so scheduling and the image pull come
out of its one-minute reserve rather than being covered by it. And the
five-minute conflict retry inside kubectl scale never runs here, since
without a precondition it sends a merge patch with no resourceVersion
to conflict on.

helm-controller's finalizer, which the hook reads to tell whether a
release being deleted is still held, is now tied to the constant the
controller uses, as the fence already was.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Alexey Artamonov 7b84919db3 fix(fluxcd): warn about a foreign suspension only on a stuck release
flux-plunger raised its SuspensionNotOwned and TerminatingWhileSuspended
Warnings on a suspension it does not own when the release carried its
processed-version annotation. That annotation is never removed, so
every later suspension of such a release, by a migration, a backup
restore or an operator, drew a Warning: one telling the reader to
resume it, which mid-restore is the one thing nobody wants, and one
calling a migration's deliberate suspend-then-delete an accident. Both
also missed the leftover they are for when the first recovery stopped
before writing the annotation.

A suspension an earlier flux-plunger left mid-recovery freezes the
release in the failure recovery started from, since helm-controller
rewrites no condition of a suspended release. Both Warnings now key on
that state instead, and the README's candidate query with them.

Two comments are qualified where they claimed more than holds: a stop
that overran the manager's grace period can still let a reconcile set
a fence after the drain's last listing, and a process stopped before
controller-runtime closes Elected holds the Lease yet does not drain.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Alexey Artamonov caeaa6b5e5 fix(fluxcd): make flux-plunger's shutdown drain report what it left
The drain's only report was an Info line, so an alert on errors never
saw fences left behind; it named no release, and printed held: -1 when
no listing had answered. A flux-plunger that never led logged that it
had released every fence next to the line saying it drained nothing.
Fences left behind are now an error naming each release, an unanswered
listing says so, and a process that never led says only that.

The drain writes its Events itself, since the manager's recorder stops
with the manager, but it created one Event object per pass for a
release it revisits every second, and each write had a deadline of its
own outside the drain's. Repeats now fold into one counted Event, as the
manager's recorder folds them, and every write is bounded by the drain.

Tests now hold the outcome line, the retry of a release whose fence
cannot be given up yet, and the pause between passes, none of which any
suite noticed when removed.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Alexey Artamonov 8bcb709ce8 fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time
A drain deadline of zero or less is already spent when the drain
starts, so flux-plunger would exit holding every uninstall fence
without listing one. The chart could produce one: its 40-second floor
was a literal, so raising the manager's stop bound or the margin before
SIGKILL, which the drain subtraction is already tied to, would leave the
floor rendering a non-positive deadline. The binary now refuses such a
flag at startup, before it reaches for a cluster, and the chart's floor
and the timings the README and the values comment state are held to
the same constants.

A whole number of a million seconds or more printed in exponent form
and was refused as a fraction; the range is now checked first, so it
gets the range message.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Aleksei Sviridkin 785cbc391b fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (#4293)
## What this PR does

Fixes #4060.

`flux-plunger` recovers a HelmRelease stuck with `has no deployed
releases` by briefly suspending it, deleting the stale Helm storage
Secret, recording a processed-version annotation, and unsuspending.
helm-controller v1.5.x skips the Helm uninstall and drops its finalizer
when it handles a delete on a suspended HelmRelease, and it wakes on the
same delete event flux-plunger does, so a delete arriving inside that
window orphaned every resource the release owns (for a VMDisk:
DataVolume, PVCs, the CDI importer pod, RBAC, WorkloadMonitor and the
storage Secret itself). Lifting the suspension on the delete event
cannot win that race reliably. The bug is not specific to VMDisk.

The change makes such a delete reach the uninstall:

- **An uninstall fence.** The suspension is applied together with a
flux-plunger finalizer, `flux-plunger.cozystack.io/uninstall-fence`, in
one server-side apply, so flux-plunger never holds a suspension without
it. If a delete arrives while the release is suspended, the fence keeps
the object whether helm-controller or flux-plunger handles the delete
first: flux-plunger lifts its suspension and keeps the fence,
helm-controller reconciles the unsuspended generation and runs the
uninstall, and flux-plunger removes the fence once helm-controller's
finalizer is gone and it has observed that generation. A recovery that
completes lifts the suspension and the fence together. A terminating
HelmRelease never begins recovery, and termination is re-checked
immediately before deleting the storage Secret.
- **Only flux-plunger's own suspension is acted on.** Ownership is
decided by which field manager owns `spec.suspend` in
`metadata.managedFields`: the apiserver transfers the field away the
moment an operator (or `cozyhr resume`, or a backup/restore controller)
writes a different value to it, so a later suspension is never misread
as flux-plunger's. A suspension set through the shared
`flux-client-side-apply` manager (platform migrations, the tenant delete
hook, an operator's manual `kubectl apply`) is never claimed. A release
deleted under another manager's suspension is that manager's decision:
flux-plunger does not hold it with the fence, and helm-controller leaves
its resources in place, as the migrations that suspend and then delete
rely on.
- **No suspension taken over in a race.** Acquiring applies without
forcing first and, on a conflict, re-fetches and forces only if
`spec.suspend` is still `false`: a backup controller owns the field at
`false` via a dynamic-client `Update`, which a non-forced apply can
never take. Every apply flux-plunger sends, acquiring as well as lifting
the suspension or the fence, is preconditioned on the `resourceVersion`
and `uid` of the read that decided on it. The `resourceVersion` makes a
write landing in between fail the apply instead of being absorbed:
acquire then defers rather than adopt or take over a suspension set
meanwhile, and lifting the suspension and the fence from a release read
as live cannot delete it without an uninstall if helm-controller has
handled a delete in the meantime. The `uid` stops an apply to a release
that has gone from creating an empty one, since server-side apply
creates what does not exist. A conflict is retried through the requeue.
The apply carries only `spec.suspend` and the fence, so it cannot leak
the required `spec.interval` and conflict on it.
- **Fresh reads.** The release, every re-check and the storage Secret
list are read from the apiserver (an uncached `APIReader`), not the
informer cache. A lagging cache would let the processed-version guard
delete the revision below the one just reclaimed, and the ClusterRole
grants secrets `get`/`list` but not `watch`, which an informer needs to
stay synced; flux-plunger deliberately does not cache Secrets. Every
step requeues with backoff on failure.
- **Events.** A Warning when flux-plunger holds a suspension it failed
to clear, and a Warning when a delete has been held by the fence for
more than ten minutes waiting for helm-controller (the held state is
re-checked on a timer, not only on helm-controller's next status write).
A release deleted under another manager's suspension gets a Normal event
when that is a migration's choice, and a Warning when the release
carries flux-plunger's processed-version annotation, because then the
skipped uninstall is an accident, not a decision. Normal events also
mark an externally-owned suspension left alone and a recovery deferred
because the release changed between read and write; an external
suspension on a release flux-plunger has recovered before is a Warning
instead, because it is most likely one an earlier flux-plunger left
mid-recovery (see Operating notes). Events from the shutdown drain are
written straight to the apiserver, since the manager's recorder stops
with the manager.
- **The fence is released when flux-plunger goes.** Once its manager has
stopped, and with it every reconcile, flux-plunger drains its fences
under a deadline inside the pod's termination grace period; a stop that
overran its 10s bound is drained too and still reported: a recovery in
progress is aborted, the suspension and the fence lifted together for
the next flux-plunger to start again, and a release being deleted has
flux-plunger's suspension lifted and keeps the fence until
helm-controller has run its uninstall. A release helm-controller does
not handle in time keeps its fence. The drain lives in the controller,
so the envtest suites hold it against a real apiserver's generations and
finalizers. A pre-delete hook on the chart is the backstop for what the
drain could not finish, so uninstalling the component never leaves a
release it cannot finish deleting: it stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and fence without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, the outcome the controller's own relinquish has, with a JSON
patch of `spec.suspend` preconditioned on the `resourceVersion` and
`uid` it read; not with `kubectl apply --server-side`, which on a
release last written by a client-side `kubectl apply` moves that apply's
fields to the applying manager and strips every one the body leaves out.
helm-controller runs a pre-delete hook under the release's uninstall
timeout, 300s by default and not set by the operator, so the Job's
deadline sits under it. A Deployment that cannot be stopped or a listing
that never succeeds ends the hook before any fence is touched, and a
fence it cannot remove ends it after the sweep; either way it exits
non-zero, because a failed pre-delete hook leaves the release untouched
and helm-controller retries the uninstall, hook included, whereas
exiting 0 would let the component go and strand the fence. A
flux-plunger pod still there after its grace period, typically one stuck
`Terminating` on a `NotReady` node, is only reported, and the pod wait
is cut to what is left of the hook's retry window; because such a pod
may still be setting fences, the fences are listed again after the sweep
and any found fails the hook. The Job runs once and is kept after
success so its report stays readable until its TTL;
`helm.sh/hook-output-log-policy` is deliberately not used, because
helm-controller never sets Helm's hook output writer and calls a nil one
for it (fluxcd/helm-controller#1562). The hook's kubectl image is
digest-pinned in the chart values, and its RBAC is only what `kubectl
scale`, the listing and the patches issue. The chart's README documents
the finalizer and how to clear it by hand.

A single writer is kept by leader election, not by the rollout. A second
process acting under the same field manager could clear a suspension the
first is still recovering, and no Deployment strategy prevents one: the
next pod starts as soon as the old one is terminating (Kubernetes v1.35
has no pod-replacement policy for Deployments), and an eviction or a
node drain replaces a pod the same way, while the old one stops and
drains. The chart now passes `--leader-elect` with a Role on `leases` in
the release namespace. The Lease is not given up on stop and outlasts
the last renewal, the renew deadline, the manager's bounded stop (10s)
and the drain, and only a flux-plunger that led drains at all, so the
next one acts only once the old one is done; the price is that a new pod
waits for the Lease to run out, a little under two minutes at the
default. The timings derive from one value, `shutdownGracePeriodSeconds`
(90 by default, refused outside 40 to 140 at render): the pod's
termination grace period, the drain deadline (25s less: the manager's
10s stop plus a 15s margin before the kubelet's SIGKILL, both tied to
the Go constants by a contract test) and the hook's pod wait. The
strategy keeps `maxSurge: 0` and `maxUnavailable: 1` and stays
`RollingUpdate` rather than `Recreate` on purpose: the chart shipped
without a strategy block, so every existing install's live Deployment
carries the apiserver-defaulted `rollingUpdate` object and
helm-controller applies server-side, where laying `type: Recreate` over
that populated block is rejected on upgrade.

### Operating notes

A HelmRelease carries the fence only while flux-plunger is recovering it
or waiting for helm-controller to uninstall it. A fenced release whose
uninstall keeps failing stays `Terminating` exactly as any HelmRelease
does under helm-controller's own finalizer, and raises the Warning
above. Stopping flux-plunger drains its fences, and uninstalling it
releases whatever the drain left through the pre-delete hook; a fenced
release that is deleted while flux-plunger is merely scaled to zero, or
killed without a graceful stop, and that helm-controller has not
uninstalled, stays `Terminating` until flux-plunger runs again. To
release one by hand, after confirming its uninstall has run or is not
wanted:

```bash
kubectl patch helmrelease NAME --namespace NAMESPACE --type json --patch '[{"op":"test","path":"/metadata/finalizers/INDEX","value":"flux-plunger.cozystack.io/uninstall-fence"},{"op":"remove","path":"/metadata/finalizers/INDEX"}]'
```

where `INDEX` is the position of
`flux-plunger.cozystack.io/uninstall-fence` in `metadata.finalizers`;
the `test` operation makes the patch fail rather than remove another
finalizer if the list has changed.

Earlier flux-plunger releases set `spec.suspend` through the shared
`flux-client-side-apply` manager, so a suspension one of them left
behind cannot be told apart from a migration's or an operator's and is
treated as external: it is not cleared automatically. Such a leftover
exists only if the old pod stopped in the middle of a recovery, and
flux-plunger raises a `SuspensionNotOwned` Warning on it. The chart
README carries this section too. To list candidates, suspended releases
carrying the flux-plunger annotation whose `spec.suspend` is not owned
by flux-plunger's apply:

```bash
kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | jq --raw-output '.items[] | select(.spec.suspend == true) | select(.metadata.annotations["flux-plunger.cozystack.io/last-processed-version"] != null) | select([.metadata.managedFields[]? | select(.manager == "flux-plunger" and .operation == "Apply" and .fieldsV1["f:spec"]["f:suspend"] != null)] | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"'
```

Clearing one is an operator action: confirm nothing else (a migration, a
restore) is holding it, then resume it with `flux resume helmrelease
NAME --namespace NAMESPACE`.

### Testing

The regression the issue asks for is
`TestEnvtest_DeleteDuringRecoveryReachesUninstall`: against a real
apiserver, a delete lands while flux-plunger holds its recovery
suspension, and helm-controller's v1.5.0 delete handling is reproduced
step for step (uninstall only when not suspended, finalizer dropped
either way, observed generation recorded). It covers both orders,
helm-controller handling the delete first and flux-plunger handling it
first; without the fence the first order leaves the release gone with no
uninstall. The same suite pins the managedFields ownership dynamics a
fake client cannot represent: the field transfer, the forced acquire
over a dynamic-`Update` owner, the `resourceVersion` precondition
rejecting a write that races either apply, the boundary that a
`flux-client-side-apply` suspension is never claimed, and interval
preservation. The fence mechanics were also checked against a live
helm-controller: with another finalizer holding a deleted, suspended
release, lifting the suspension made helm-controller run the uninstall
and remove the release's resources.

The envtest suite runs on every pull request: a new `make
test-controllers-envtest` target builds a pinned `setup-envtest` and
fetches a kube-apiserver and etcd into `_out/envtest`, which CI caches
and the target reuses without asking the release index, takes the
HelmRelease CRD from the in-tree Flux manifests, and CI calls it next to
`make test-controllers`; the control plane gets a longer start timeout
because it starts beside the other test goals. The drain is held there
too: it aborts a live recovery, carries a delete through the uninstall
and lifts the fence only once helm-controller has observed the
unsuspended generation, keeps the fence when helm-controller never
answers, releases a fence left without a suspension at once, and leaves
another manager's suspension alone.
`TestEnvtest_TheNextFluxPlungerLeadsOnlyAfterTheDrain` runs two real
managers on one Lease: a clean stop still drains, and the second manager
becomes leader only after the first one's drain has ended; it goes red
if the Lease is given up on stop. Unit tests cover the controller logic
(every branch, ordering, requeue, and the ownership decision through an
injected seam), the Lease timing relations, that only a leader drains
and through the manager's API reader rather than its stopped cache, that
a failed listing is retried, that a stop which overran its grace period
is still drained and its events reach the apiserver, and contract tests
tie `main.go`, the chart's flags, the drain's arithmetic and the hook's
names to the Go code. The pre-delete hook is covered by helm-unittest
and by `hack/flux-plunger-fence-hook_test.bats`, which runs its rendered
script against a fake `kubectl` that keeps each release's finalizers,
suspension owner, generation and observed generation, bumps the
generation on a spec write as the apiserver does, answers NotFound
without `--ignore-not-found`, spends simulated time on waits, lists a
release once however many copies of the fence it carries, refuses
`kubectl apply`, lifts a suspension only through the preconditioned JSON
patch and never one another manager also holds, reads a negative
`kubectl wait` timeout as a week as kubectl does, and applies a
finalizer removal only when it is the guarded test-then-remove at an
index that holds the fence. Every mutation in a set covering each
predicate conjunct, each retry, each exit path, the re-listing, the
timings and the selector turns the hook suite red, as does every
mutation in the Go set covering the drain's branches, the Lease timings
and the events; three drain mutations that still survive are tracked in
#4782. The hook script was also run as a Job in the hook's own image on
a live cluster, succeeding, with the Deployment already gone, and with
patch denied by RBAC, where it exited non-zero and named the release it
left; that run caught that a per-call `--request-timeout` makes kubectl
drop its in-cluster config, so the hook sets none and the fake rejects
it. A later live run caught `kubectl apply --server-side` stripping the
finalizers and spec of a release created by a client-side apply; with
the JSON patch the same releases kept every field, the suspension
flux-plunger alone held was lifted, a co-owned and a foreign one stayed,
and every fence went.

### Remaining limitations

- A delete that arrives after recovery has deleted the storage Secret of
a release whose only revision was that one finds no revision to
uninstall; helm-controller then treats the release as already gone.
- A writer that sets `spec.suspend: true` while flux-plunger already
holds it at `true` does not take the field over, because the value does
not change. If such a writer then deletes the release inside the
recovery window (the suspend-then-delete that the tenant delete hook and
some platform migrations use to skip the uninstall), flux-plunger still
sees its own suspension, lifts it, and helm-controller runs the
uninstall that writer meant to skip. The same held before this change,
when the end of recovery unsuspended unconditionally.
- The already-processed guard compares revision numbers only. When Helm
re-creates a revision under the same number after recovery deleted it (a
release whose only revision fails again the same way), recovery runs
again for it.
- `listHelmReleaseSecrets` reads the local cluster only, so a
HelmRelease that targets a remote cluster via `spec.kubeConfig` (whose
storage Secrets live in the remote cluster) is not recovered; it is
skipped, never mis-handled.
- A new flux-plunger pod waits for the old one's Lease to run out before
it acts, on every restart and after a crash: a little under two minutes
at the default. Recovery is not latency-sensitive, and this is what
keeps a single writer while the old pod drains.
- The leader Lease is left in the release namespace after the chart is
uninstalled, as with any controller-runtime leader election; it is
inert.

### Screenshots

Not applicable: no UI change.

### Downstream repositories

The trigger map was walked against the diff: it touches
`internal/controller/fluxplunger`, `cmd/flux-plunger`, the
`packages/system/flux-plunger` chart (templates including RBAC for the
leader Lease, a pre-delete hook, values, tests, its package `Makefile`,
README), a new bats test under `hack/`, the root `Makefile` (a new
`test-controllers-envtest` target; existing targets are unchanged) and
the pull-request workflow that calls it and caches its binaries. No row
matches: ccp gates on existing `hack/` files and make targets, and
nothing there is moved, renamed or changed in behaviour; a new test file
and a new target reach none of them. The website's FluxPlunger section
describes the recovery steps, which still hold; it does not describe the
fence or the drain, which the chart README now documents.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(fluxcd): a HelmRelease deleted while flux-plunger is recovering it now reaches the Helm uninstall instead of leaving its resources (DataVolume, PVCs, RBAC, storage Secret) behind
```
2026-10-12 15:23:49 +03:00
Aleksei Sviridkin 8deaa8e14c fix(vm-disk): name the importer error in the disk's WorkloadsReady message (#4815)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A VMDisk whose HTTP source answers 404 or does not resolve never
finishes importing, and since #4380 its `WorkloadsReady` condition says
only `DataVolume vm-disk-x is ImportInProgress`. That reads the same as
a large download that is still running. CDI fails an import only on a
fatal error (a checksum mismatch) and restarts the importer pod
otherwise, so the phase never moves, and the reason the disk will not
finish was visible only in the importer logs.

CDI already keeps that reason on the DataVolume: the `Running` condition
goes `False` with the importer's termination message, and keeps it
through the back-off between restarts. The WorkloadMonitor now appends
it, so the condition reads `DataVolume vm-disk-x is ImportInProgress:
Unable to connect to http data source: expected status code 200, got
404. Status: 404 Not Found`. The console already shows the
`WorkloadsReady` message on the disk's overview page, so no UI change is
needed. The same text covers an unschedulable importer, a failed image
pull and a checksum failure, since CDI reports all of them through
`Running=False`.

Only `Running=False` is read: `Unknown` is not a failure, and `Bound`
repeats the same error behind PVC details. For a host-assisted clone CDI
joins the target and source pod messages as `<target> and <source>`,
which comes out as ` and ` when both pods are still being created, so
empty halves are dropped.

Behaviour to be aware of: while the importer pod is briefly running
between restarts, CDI flips `Running` to `True` and clears the message,
so the condition shows the bare phase for those seconds. This is not a
terminal state either: a disk with a broken source still stays
`ImportInProgress` and keeps retrying. Deciding when such an import
should count as failed is left for a separate change.

### Tests

A table test covers a 404, an unresolvable host, an unschedulable
importer, a fatal checksum error, a DataVolume without a phase and
joined clone messages with one or both halves empty, plus the cases that
must not add text: an importer that has not started, a running importer,
an `Unknown` condition, a populated disk and an error that only sits on
`Bound`. Each conjunct of the condition check and the append itself were
mutated and each mutant fails the test. A second test runs against a
DataVolume captured from a test cluster after its importer had restarted
about two thousand times on a 404, with only names and the URL changed,
so the parsing is pinned against what CDI actually writes rather than a
hand-built shape.

### Screenshots

No UI code changes: the console renders the `WorkloadsReady` message it
already receives. A message that carries a long source URL is not
wrapped on the overview card; that is a styling follow-up for the
console and is left out of this change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only non-Go change is a sentence in the
`packages/apps/vm-disk/README.md` notes; no values, schema, API kind or
default changes. The website reference page for vm-disk is regenerated
from that README by the release bot.

### Release note

```release-note
fix(vm-disk): the WorkloadsReady condition of a VM disk now includes the importer error reported by CDI, such as a 404 or an unresolvable host, so a disk whose source keeps failing shows why it is stuck instead of only ImportInProgress.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Workload status messages now include available DataVolume import
failure details, such as HTTP errors or DNS resolution problems, making
the cause of an import issue clearer.
* **Documentation**
* Updated the workload readiness documentation to explain when
DataVolume phase and import error details appear in status messages.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 12:29:50 +03:00
Aleksei Sviridkin d9689bd6e7 fix(gateway): stop describing Cilium 1.19.5 as the shipped release
With Cilium at 1.20.2 the tenant gateway comments, README and one
status message still explained behaviour as that of "the pin" at
1.19.5. Several of those claims no longer hold: a Gateway with two
routed passthrough ports now gets an Envoy listener per port, Cilium
judges a route's kinds against the listener the route names, and the
shipped operator accepts an HTTPS and a passthrough listener on one
port when the passthrough hostname is strictly narrower.

Behaviour that is unchanged is re-anchored on the 1.20.2 source,
behaviour that is gone is stated as history, and the refusal of one
hostname on two passthrough ports no longer blames a Cilium limitation
the shipped release does not have.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin ad3f15d4f3 feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.

The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.

Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.

The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.

The backend resolution, its watches and the RBAC rules that served them
go away.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin bdca165f4c feat(gateway): report a GatewayClass that does not serve TLSRoute
A GatewayClass lists the features it serves in status.supportedFeatures,
and TLS passthrough is TLSRoute. On a class that leaves it out the
passthrough listeners are accepted and serve nothing, so the
TenantGateway now reads Ready=False/GatewayClassUnsupported naming the
class instead of Ready. A class that lists no features at all is not
judged: implementations predating the field leave it empty.

The class writes that list after it is accepted and may change it when
its controller is upgraded, so a GatewayClass event requeues the
TenantGateways on it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:25 +03:00
Aleksei Sviridkin d624049a3a feat(gateway): allow tlsPassthroughListeners under dns01 and existingSecret
The refusal existed because Cilium 1.19.5 served every Gateway port
from one Envoy listener matched by SNI alone, so the "*.<apex>"
terminate listener of the wildcard modes and a passthrough listener
under the apex could reach each other's backend. The shipped Cilium
1.20.2 gives each port its own Envoy listener when SNI overlaps across
ports, and Gateway API treats listeners on different ports as
compatible, so the pair is served as declared. edge still refuses the
field: it renders no TLS listener for an entry to bind to.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:53:25 +03:00
Alexey Artamonov 970ccc2578 test(vm-disk): pin the importer error against a real stuck DataVolume
The table test builds its Running conditions by hand, so it can only
confirm the shape it already assumes. A DataVolume read off a cluster
where the importer had failed on a 404 for days carries a Bound
condition with the same error, which the message must not pick up, and
proves the error sits where the controller looks for it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-08 09:46:36 +03:00
Alexey Artamonov 0b9a801a2b fix(vm-disk): name the importer error in the disk's WorkloadsReady message
A disk whose HTTP source answers 404 or does not resolve never leaves
ImportInProgress: CDI fails an import only on a fatal error and restarts
the importer pod otherwise, so the condition only ever said the disk was
still importing. The reason it will not finish was visible only in the
importer logs and on the DataVolume, whose Running condition carries the
error through the back-off between restarts.

Appending that message lets the user see why the disk is stuck from the
application status and the console, which already shows the
WorkloadsReady message for a disk.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-08 09:46:36 +03:00
Aleksei Sviridkin 2eeaa59055 test(controller): seed the DataVolume informer for controller-runtime 0.25
controller-runtime 0.25 made FakeInformer's sync state a private
channel, so the DataVolume watch tests no longer compile: they toggled
the exported Synced field and its lock. FakeInformers also builds every
informer it creates already synced, so an unsynced informer has to be
seeded into InformersByGVK and marked synced with Synced() later.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 22:29:03 +03:00
Aleksei Sviridkin bf03c84711 chore(deps): move to the Kubernetes 1.37 libraries without the apimachinery fork (#4473)
## What this PR does

This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.

The four commits are meant to be read one by one, and each builds and
passes the tests on its own:

1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.

Commits 2 and 4 are where the review effort goes; 1 and 3 are small.

Known changes in behaviour:

- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.

The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 21:45:51 +03:00
Aleksei Sviridkin ace69cb075 docs(controller): say what a nil DataVolume reader means
The field comment still said DataVolumes do not count while the reader
is nil, and that the reader appears once the kind is served. Neither
holds: the reader is installed only after the informer has synced, and
until then a monitor keeps its stored verdict, which still decides
Operational.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:38 +03:00
Aleksei Sviridkin a6a32d79cd fix(controller): recompute stored DataVolume verdicts once the reader is in
A reconcile that runs before the DataVolume watch starts finds no
reader, keeps the monitor's stored verdict and is not requeued. The
watch's initial replay then delivers only the DataVolumes that still
exist, so a monitor whose DataVolume was deleted while the controller
was down kept reporting it not ready until the periodic resync, hours
later.

Installing the reader now queues every monitor that holds a DataVolume
verdict, through the controller's own queue, so each is recomputed
against the synced cache. Monitors without a verdict are left alone:
the replay already reaches every DataVolume they could select. If the
monitors cannot be listed, the next discovery poll retries.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:37 +03:00
Aleksei Sviridkin 694f4f2ff9 fix(controller): report a disk's PVC Workload by its DataVolume
CDI copies the DataVolume labels onto the PVC it creates, so the vm-disk
monitor also publishes a Workload for that PVC, and its readiness was
bind state alone. Mid-import the PVC is already Bound: the Workload read
operational while the monitor, and the Application's WorkloadsReady,
said the disk was not populated.

A PVC whose controller is a DataVolume the monitor finds not ready now
reads not operational. While the DataVolumes cannot be read, its
Workload keeps the verdict it has, as the monitor does.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:36 +03:00
Aleksei Sviridkin 366b5e80f4 perf(controller): skip the monitor status write when nothing changed
While the DataVolume watch syncs, every WorkloadMonitor of every kind
is requeued every few seconds, and an informer that cannot sync keeps
that window open. Each pass wrote status unconditionally, so a cluster
with many monitors sent the API server a steady stream of writes that
changed nothing. Comparing against the stored status first leaves the
requeue as cache reads only.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:36 +03:00
Aleksei Sviridkin dd72583dfd fix(controller): record an event when DataVolumes cannot be read
A failed DataVolume List keeps the monitor's last verdict and retries,
and the only trace of it was a controller log line. A monitor with no
verdict yet then reads operational while its disks are unreadable, with
nothing a tenant can see saying why. A Warning event on the monitor
makes the failure visible where the bucket metrics failure already is.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:36 +03:00
Aleksei Sviridkin beeb7994cc fix(controller): start the DataVolume watch source only once
A controller-runtime Kind source adds an event handler to the informer
when it starts and never removes it; cancelling it on a sync timeout
only ends its context. Building a new source on every failed sync
attempt therefore left one more handler on the DataVolume informer per
retry, each enqueueing the same events, until the controller restarted.

The source is now started once and later attempts only wait for the
informer to sync. Because the started source keeps delivering events
between attempts, the syncing flag stays set until the reader is
installed, so a reconcile queued in that gap is requeued as well.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:35 +03:00
Aleksei Sviridkin a9b276f99b fix(controller): requeue monitors reconciled while the DataVolume watch syncs
Starting the DataVolume source replays every DataVolume to its
handler at once, and the handler queues the matching monitors, while
the reader is installed only after the source has synced. A monitor
reconciled in that window saw no reader, kept its last verdict and
returned with nothing left to queue it again, so a disk that failed
or finished while the controller was down could keep a wrong
WorkloadsReady until the next unrelated event.

Such a reconcile still publishes the replica counts and now asks to
be requeued after a few seconds, until the reader is in place or the
sync attempt gives up. It is a requeue rather than an error because
the window is expected: an error would count as a reconcile failure
and log a line for every monitor in the cluster while it lasts.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:35 +03:00
Aleksei Sviridkin 99f3d3b2b7 feat(controller): tag a DataVolume cause with a condition reason
WorkloadsReady=False covers different situations: a stopped VM has
no pod and its monitor reports it not operational on purpose, while a
disk whose DataVolume never populates is broken. A consumer that wants
to react to one and not the other had only the message text to go by.

The WorkloadMonitor status gains a Reason next to Message, set to
DataVolumeNotReady by the DataVolume check, and the Application API
carries it into the WorkloadsReady condition in place of the generic
reason whenever a monitor that is not operational names one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:33 +03:00
Aleksei Sviridkin 81ffcda86f fix(controller): keep publishing monitor status when DataVolumes are unread
The DataVolume List sits on the reconcile path of every
WorkloadMonitor, and returning its error before the status write froze
the replica counts and Operational of a monitor that selects no
DataVolume at all whenever CDI's API hiccupped.

The status is now written either way, with the last DataVolume verdict
kept whenever the DataVolumes could not be read, and the error still
returned so the request is retried. The same applies while no reader
is installed yet, so a stuck disk does not read as ready for the
window between a controller restart and the DataVolume informer
syncing.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:32 +03:00
Aleksei Sviridkin 48ece79ffc feat(controller): name the DataVolume that keeps a disk not ready
For a vm-disk, WorkloadsReady is the one condition that reports
whether CDI populated the disk, and its generic "One or more workloads
are not operational" gives an operator nothing to act on: a disk has
no workloads, and the stuck DataVolume and its phase are exactly what
they need to know.

The WorkloadMonitor status gains a message that the controller fills
with each selected DataVolume that is not ready and its phase, and the
Application API uses the messages of the monitors that are not
operational in WorkloadsReady instead of the generic text whenever one
of them names a cause.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:02:32 +03:00
Aleksei Sviridkin bf7e8f5bbb fix(controller): count DataVolume readiness in WorkloadMonitor
A vm-disk application reported its workloads operational whatever
state its DataVolume was in, so an import or clone that never
finished left the disk looking healthy while it was unusable. Its
monitor has minReplicas 0 and only pods fed Operational.

The monitor now also lists the DataVolumes its selector matches and
reports not operational while any of them is in flight, Failed,
Unknown or without a phase. UploadReady counts as in flight: CDI
holds it from the moment the upload server is ready until the
transfer succeeds, so it covers an empty disk and a half-uploaded
one alike. Waiting for the first consumer counts as ready, since
that is where such a disk rests, and so does a phase this
controller does not know, so a CDI release that renames one cannot
turn working disks red.

This is deliberately not a Helm wait: a readiness gate holds a
helm-controller worker for its whole window, and a disk that never
populates would keep taking it back. The signal lands in the
application's WorkloadsReady condition instead, and Ready keeps
following the HelmRelease.

CDI ships with the IaaS bundle only and can be installed while this
controller is running. A watch on a kind whose CRD is absent keeps
the controller from starting, so the DataVolume watch is added from a
runnable that polls discovery until the kind is served. It starts on
the first reconcile, the earliest point at which a new watch is
started at once, rather than on the next tick, so a disk created
just after a restart is not read as populated for a poll interval.
The reader is installed only once that watch has synced within a
bound: a reconcile context has no deadline, and a List on an
informer that never syncs would park every WorkloadMonitor reconcile
for good.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 14:01:13 +03:00
Aleksei Sviridkin 2df29639df fix(clickhouse, workloadmonitor): add the cluster domain to keeper hosts and stop a PVC nil dereference (#4608)
## What this PR does

Two small bug fixes, one per commit. Both were first proposed by
@officialasishkumar in #2334 and #2336. Those PRs went stale, and part
of each was fixed on main in the meantime, so this carries only what is
still broken.

ClickHouse Keeper hosts rendered without a cluster domain, as
`chk-<name>-keeper-cluster1-0-0.<namespace>.svc.` with nothing after
`svc.`. The zookeeper block in `clickhouse.yaml` redeclared
`$clusterDomain` from `.Values.clusterDomain`, which the chart never
defines, and that shadowed the value read from `_cluster.cluster-domain`
at the top of the template. Keeper is enabled by default, so every
ClickHouse release got these hosts. The keeper name test set
`clusterDomain` itself and hid the bug. It now sets only
`_cluster.cluster-domain`, like the platform does, and fails without the
fix.

The WorkloadMonitor controller panicked on a PVC with no
`storageClassName`, which is any PVC that takes the cluster default
class. The check used `||` where it needed `&&`. controller-runtime
recovers the panic, but the reconcile of that monitor fails on every
retry and never reaches its status update. A PVC with an empty
`storageClassName` also got a resource key with an empty class name
instead of `default`. A new table test covers unset, empty and explicit
classes.

The duplicate `template` key in the ClickHouse backup CronJob from #2334
and the `Operational` status fix from #2336 are already on main, so they
are not in this PR.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(clickhouse, workloadmonitor): ClickHouse Keeper node hostnames now include the cluster domain, and the WorkloadMonitor controller no longer panics on a PVC without a storage class.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Workload resource labels now use the default storage class when a
PVC’s storage class is unset or empty, while continuing to reflect
explicitly configured storage classes.
* ClickHouse ZooKeeper hostnames now consistently use the configured
cluster domain, or `cozy.local` when no domain is set. This helps
hostnames align with the cluster’s domain configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-02 13:58:29 +03:00
harshitrwt ffb6564f8b fix(controller): drop stale bucket-class label from BucketClaim labels before applying validated value
maps.Copy(workload.Labels, bc.Labels) can bring in a workloads.cozystack.io/bucket-class label directly set on the BucketClaim, which would survive unmodified when spec.bucketClassName is empty or invalid, leaving the Workload reporting a class that does not match the source BucketClaim.

Signed-off-by: harshitrwt <harshit.rawat1804@gmail.com>
2026-10-01 00:03:05 +05:30
harshitrwt f41a500b07 test(controller): cover invalid bucket class label in reconcileBucketClaimForMonitor
A BucketClassName longer than 63 chars (valid as a Kubernetes object name, up to 253 chars) is not a valid label value, so the bucket-class label is skipped with an Info log rather than failing the Workload write. This can happen with SeaweedFS pool class names (<namespace>-<pool>-lock) when the tenant and pool names are both long -- a missing label in that case does not mean no class was set.

Signed-off-by: harshitrwt <harshit.rawat1804@gmail.com>
2026-09-30 23:43:28 +05:30
harshitrwt 882929ff86 fix(controller): skip bucket-class label when BucketClassName is not a valid label value
Signed-off-by: harshitrwt <harshit.rawat1804@gmail.com>
2026-09-30 23:17:36 +05:30
harshitrwt 0c37fff5f4 feat(controller): expose bucket class on BucketClaim Workloads
Signed-off-by: harshitrwt <harshit.rawat1804@gmail.com>
2026-09-30 23:17:36 +05:30
Aleksei Sviridkin 8b8d4115fc fix(workloadmonitor): stop panicking on a PVC without a storage class
The storage class check used || where it needed &&, so a PVC with no
storageClassName (one that takes the cluster default) was dereferenced
as a nil pointer. controller-runtime recovers the panic, but the
reconcile of that WorkloadMonitor fails on every retry and never
reaches its status update. A PVC with an empty storageClassName
produced a resource key with an empty class name instead of falling
back to "default".

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-30 14:17:50 +03:00
Mattia Eleuteri e510d49b3d feat(monitoring): re-render node agents when a monitoring label changes
The node fluent-bit builds its tenant routes from a lookup of namespace
labels, and helm-controller upgrades a release only when its chart or
values digest changes. A tenant turning monitoring on therefore got no
route, and a removed one kept its route, until an unrelated change
upgraded monitoring-agents.

The reconciler digests the label across all namespaces and requests a
forced upgrade of the release when the digest moves. It digests every
labelled namespace rather than repeating the chart's selection, so it
cannot drift from the chart; the cost is an occasional upgrade that
renders the same config.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:10:55 +02:00
Aleksei Sviridkin e62f510faf chore(deps): move the Kubernetes libraries to v0.37.1
The root module's k8s.io/* move from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, its release for Kubernetes
1.37. Two things held them back: the apimachinery fork, which had to be
rebased onto every minor, and the watch handler that from
k8s.io/apiserver v0.36 panicked on the initial-events-end bookmark the
aggregated storages sent to plain watches. Both are gone.

The rest of the upgrades are what these releases require: cel-go
v0.29.2, the etcd client v3.7.0, kube-openapi, structured-merge-diff and
the x/ modules. code-generator moves with the libraries.

Regenerating the strategy CRDs that embed core types adds the
Kubernetes 1.37 fields and follows one removal: the alpha PodSpec field
workloadRef became schedulingGroup upstream. Once the new CRD is
applied, the apiserver prunes workloadRef from every stored strategy it
serves, and the object's next write erases it from etcd. It only ever
took effect behind an alpha feature gate.

The gateway README and the tenantgateway CRD tests cite the
apiextensions-apiserver list-type ratchet at the pinned version; it is
byte-identical in v0.37.1, so only the version they name changes.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-26 16:08:25 +03:00
Aleksei Sviridkin 2c2c0301f7 chore(deps): roll up pending Renovate updates and move to Go 1.27.1 (#4447)
## What this PR does

This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.

Before the bumps:

- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.

Held back on purpose, with the reasons in the commit messages:

- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.

govulncheck reports nothing reachable in any of the five modules,
standard library included.

Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.

Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.

Closes #4487.

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added a restore resource policy option for Velero strategies.
  * Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
  * Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-26 10:17:29 +04:00
IvanHunters 0b7dd23b99 fix(appdef): let one ApplicationDefinition own each application kind
Nothing enforced that spec.application.kind is unique, and each consumer that
maps a kind back to a definition resolved a duplicate its own way: the
chartRef reconciler let every definition rewrite every release of its kind,
the lineage webhook and the CA projection reconciler took whichever came
first in an unordered list, and cozystack-api registered one resource per
definition. A second definition declaring a kind already in use could
therefore repoint that kind's tenant releases at its own artifact.

Add one ownership rule, the definition created first owns the kind with ties
broken by name, and route every consumer through it. A definition that
arrives later with a kind already in use no longer manages that kind's
releases whatever it is named: the chartRef reconciler skips it with a
Warning event and re-reconciles the next owner when the current one goes,
cozystack-api registers the owner only and logs the rest, its restart hash
moves when ownership moves, and the lineage and CA projection lookups
resolve the owner.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-25 09:39:35 +03:00
Aleksei Sviridkin 7a0aab2bdc refactor: apply the Go 1.27 modernizers
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.

Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 17:23:18 +03:00
IvanHunters 1f9bad0643 fix(workloadmonitor): populate VM instance-profile label from the KubeVirt preference annotation
getWorkloadMetadata filled the workloads.cozystack.io/kubevirt-vmi-instance-profile
label from the annotation kubevirt.io/cluster-instanceprofile-name, a key KubeVirt
never stamps. KubeVirt records the cluster preference (which carries the guest OS
profile) under kubevirt.io/cluster-preference-name, so the profile label was never
emitted and a VM's instance profile never reached the billing metric. The sibling
instance-type label worked only because its key is the real one.

Read the correct annotation. The emitted label name is unchanged, so the downstream
metric pipeline is untouched.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-24 17:18:06 +03:00
Aleksei Sviridkin 626e1fe19d test(gateway): pin that a second reconcile leaves cert-manager objects alone
Idempotency was pinned only for the Gateway and the HTTP01 Issuer. A
cert-manager type change that makes the rendered spec differ from the
stored one would turn every reconcile into an update. Cover the DNS01
Issuer and wildcard Certificate and the per-listener HTTP01 ones.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:03 +03:00
Aleksei Sviridkin 5c857c9d9f test(kube-ovn): pin the plunger's OVN cluster metrics
The metrics had no tests. Pin the series written for a cluster and its
members, and that dropping one cluster removes only its series.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:03 +03:00
IvanHunters 22f2d76ead test(controller): pin the invalid-annotation and kind-selector paths of the wait carry-over
The reconciler skips a disable-wait annotation it cannot parse, and it
selects releases by application kind. Removing either left the suite
green; each now has a test that fails without it.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-23 20:18:06 +03:00