Commit Graph
607 Commits
Author SHA1 Message Date
myasnikovdaniil 970191dc8a fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does

A ClickHouse `RestoreJob` restored the newest archive under the app
prefix, whichever `Backup` it named. Strategy pod generated the archive
name with `date -u` and nothing was recorded on the `Backup`, so restore
could only list the remote archives and take the last one. Restoring an
older `Backup` brought back the latest data.

Now the controller names the archive before it creates the backup Job
(`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so
retried pods write the same one), passes it to the pod as `.ArchiveName`
and records it in the `Backup` `driverMetadata`. Restore takes exactly
that archive and fails with its name when storage has no such archive.
The name is recorded only when the Job's pod got it, so a Job left from
the previous controller during an upgrade, or a custom strategy with its
own names, leaves no name instead of a wrong one.

A retried pod reuses the name, so it waits for an attempt the sidecar
still runs (no 423) and asks for `resume`.

A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after
the schedule slot, so a `BackupJob` created again under a used name met
the earlier run's `Backup` when its Job completed, took it as its result
and left the archive it just wrote recorded nowhere. Now it fails if
that `Backup` records another archive, and the message names both. A
retry of the same run still gets its own `Backup` back.

Old `Backup`s have no recorded name and stay restorable: the pod logs a
`WARNING` and takes the newest archive stamped no later than `takenAt`,
and fails when nothing qualifies. While the chart rolls out, previous
controller pod can still render the new template and pass no `takenAt`,
so a restore of a `Backup` without recorded name fails before it touches
any table, and works when created again after the rollout.

Fixes https://github.com/cozystack/cozystack/issues/5199

Tests are go unit tests for naming, recording, a reused `BackupJob` name
and restore render, and `hack/altinity-backup-strategy.bats`, which runs
the shipped script against a stubbed clickhouse-backup API (the restore
tests go red on the old template). Checked on a live ClickHouse with two
backups of 1500 and 2000 rows: restore of the older one by recorded name
ends with 1500 rows, fallback for an unnamed older backup picks the same
archive with the warning, a missing archive fails without touching the
table, a retry over an existing local backup completes under the same
name. The controller binary itself was not run on the cluster, only its
rendered output in the pod.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backups now use a consistent archive name tied to the backup job, and
completed backups retain that name for future restores.
* Restores can select the recorded archive and backup timestamp. If no
archive was recorded, the system selects the newest eligible archive for
the application.
* **Bug Fixes**
* A restore now fails when its recorded archive is missing instead of
silently selecting a different one.
* Backup requests avoid starting a duplicate action when the matching
action is already in progress.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-06 11:54:38 +05:00
Myasnikov Daniil d7ae34dfc3 fix(backups): fail a ClickHouse BackupJob whose Backup name is taken
A Backup outlives the BackupJob that made it, so a BackupJob created
again under a name already used meets the earlier run's Backup when its
own Job completes. That happens by hand, and from a Plan reconciled
again inside its starting deadline, since a Plan names each BackupJob
after its schedule slot. The Altinity driver took the earlier Backup as
its result: the BackupJob turned Succeeded with a BackupRef recording
the earlier run's archive, the archive this run wrote was recorded on
nothing, and restoring what the BackupJob reported brought back the
older data.

An existing Backup is now taken as this run's only when it records this
run's archive, which is what an earlier reconcile of the same run
leaves when its status write is lost. Any other fails the BackupJob,
naming both archives, and leaves the earlier Backup untouched.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-05 07:56:44 +00:00
Myasnikov Daniil 59f0013b1f fix(backups): attribute a MariaDB dump only when no other run can own it
Every backup of a MariaDB application writes its dump under one shared
prefix, so the probe took the dump whose file-name timestamp (the
backup pod's clock) and LastModified (the object store's clock) both
fall in the operator Backup's lifetime, with a minute of slack for
skew. That keeps a neighbour's dump out only while this run's own dump
is in the window. With the pod clock more than the slack behind, the
own dump is named before the window opens, and a previous run that
finished within the minute before this one was created becomes the
single match: a 123-byte dump was reported as 666 bytes. A pod clock
ahead or an object store clock behind does the same with the next run,
and an own dump that is gone by the time it is looked up, as it can be
for a restore long after the backup, leaves a neighbour as the single
match with no skew at all.

A dump is now attributed only when ownership is proven:

- it is the one dump that either clock places in the window. A dump
  that one clock places there and the other does not may be this run's
  own with a skewed clock, so it leaves any other match unproven, the
  way a second dump in the window already did, and is not taken itself;
- both of its clocks place it in the window;
- neither clock places it in the window of another operator Backup
  that writes under the same bucket and prefix and has completed. This
  rejects a neighbour's dump when this run's own is missing, which the
  clocks alone cannot tell.

Operator Backups still running are not counted as neighbours: their
window has no end, and one stuck short of Complete (a Job whose pod
never starts stays JobRunning for good) would otherwise keep every
later run of the application from being sized. A dump such a run
writes inside this run's window still competes with this run's own.

Narrower slack does not prove ownership: a neighbour that finished
within the narrower slack still matches, and a smaller skew pushes the
own dump out. The Job's start and completion times are stamped by the
control plane, like the operator Backup's own, so they narrow the
window by seconds without removing the dependence on the two clocks a
dump carries. What is left: a neighbour's dump is taken only when this
run's own is missing from the listing or misplaced on both clocks and
the neighbour's operator Backup no longer exists or has not completed.
Every other unproven case leaves the size unset, as two runs that
overlap or follow each other within the slack already did.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:24:09 +00:00
Myasnikov Daniil 560c44f71e fix(backups): read a MariaDB dump from where its operator Backup wrote it
The size probe took the bucket, endpoint, prefix, credentials and TLS
settings from the strategy as it renders on the reconcile that creates
the Backup, which is not necessarily what it rendered when the operator
Backup was created: an existing operator Backup is reused untouched,
and a backup can run for up to the driver's 30-minute deadline. An
admin who edits the strategy in between sends the probe to the new
location, where it measures the wrong prefix or nothing at all.

The operator Backup's own spec.storage.s3 is where the dump went, so
the probe reads that. It is also the only source a restore can use,
since it targets that operator Backup long after the strategy may have
changed. An operator Backup that does not store to S3 is refused before
anything is listed.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:18:39 +00:00
Myasnikov Daniil 21f51a2e74 refactor(backups): return the matched MariaDB dump, not only its size
A MariaDB restore has to name the dump it restores from: without a
target time the operator restores the newest dump under the prefix,
which is not necessarily the one of the Backup being restored. The size
probe already finds that dump, but it returns only its size, folds a
missing, an ambiguous and an unreadable listing into the same 0, and
lives on BackupJobReconciler, which the restore reconciler is not.

Split the probe in two: findMariaDBDump reads the bucket and returns the
matched object (key, file name, the timestamp in its name, size and
LastModified) or an error that tells "not found" from "ambiguous"; the
size path keeps its timeout and reads only the size. The Secret lookup
and the S3 client move onto an objectStore value that any reconciler
can build from its own client, and BackupJobReconciler keeps one-line
wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly
as before.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:16:15 +00:00
Myasnikov Daniil c140354977 test(backups): build operator Backups with the operator's success shape
mariadb-operator marks a Backup Complete=True for a failed Job as well
as a successful one; only the reason tells them apart (JobComplete with
message Success, or JobFailed with message Failed). The fixtures that
stand for a finished backup set Complete=True and nothing else, so they
describe a state the operator never produces and would read as a
failure to a driver that checks the reason. Give them the success shape
the operator writes.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 13:13:54 +00:00
Andrey Kolkov 92fa979690 fix(backups): trust only the endpoint CA where the backup writer does
With an endpoint CA the probe trusted the system roots plus that CA.
barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the
default trust store, so a CNPG probe could accept an endpoint the backup
itself would have refused. Let the connection say whether the CA adds to
the system roots or replaces them, and replace them for CNPG.
mariadb-operator appends its CA to the system pool, so MariaDB keeps
that.

Pin in reconcile tests that a Create the apiserver rejects still fails
the BackupJob, so only the read-back of an existing Backup is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:23 +00:00
Andrey Kolkov 080df1b2f2 fix(backups): surface a Backup that cannot be read back and skip its probe
A Backup that exists but cannot be read back is retried through the
controller's backoff, which left the BackupJob Running with nothing on
it saying why. Set Ready=False with reason BackupNotReadBack before
handing the error back.

When the check for an existing Backup itself fails, the Backup may well
exist, so a size probe would only be discarded, and a read that keeps
failing reran it, up to the probe timeout on the single BackupJob
worker, on every retry. Skip the probe in that case and create the
Backup without a size.

Drive both behaviours through the MariaDB and CNPG reconciles in tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 523d74a503 fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and
the read that follows fails, the drivers failed the BackupJob, although
the backup and its Backup both exist. Return a distinct error instead,
which the drivers hand back to the controller so the reconcile is
retried with backoff and then finds the Backup.

minio.New already refuses an endpoint with a path prefix and accepts a
trailing slash, so drop the duplicate check in the shared client; the
CNPG probe keeps its own check because it parses the URL itself and
would otherwise drop the path before minio sees it.

Say in the sizeBytes API doc what the value covers, such as the base
backup without the shared WAL archive for CNPG, and that strategies
which cannot attribute stored bytes to one backup leave it unset.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 2b21763577 fix(backups): do not fail a backup on a transient retry-check read
The check for a Backup an earlier reconcile already created returned
its read error, and the drivers turn any error from that path into a
terminally Failed BackupJob. An apiserver timeout right after the
operator backup completed therefore lost a backup whose data was safely
in S3. The check only saves a probe, and the Create that follows already
handles an existing Backup, so log the error and go on to Create.

A trailing slash on the S3 endpoint is not a path prefix; the operators
accept it, so do not refuse it.

Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB
window and the probe timeout with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov a3d854739d fix(backups): attribute MariaDB dumps on two clocks and read retries uncached
The MariaDB probe identified this run's dump by the timestamp in its
file name, which comes from the backup pod's clock. With that clock
skewed past the slack, this run's dump falls outside the window while a
neighbouring run's dump can be the single match left, and its size is
reported as this backup's. A dump now has to fall inside the window by
both its name and its LastModified, which the object store stamps, so a
single skewed clock leaves the size unset instead.

The name parser also accepts the layout later mariadb-operator releases
write for compressed dumps, backup.<ts>.sql.<ext>, and checks the
compression token in either layout, so an operator upgrade does not
silently stop the measurement.

The retry check and the AlreadyExists refetch read the Backup through
the cache. A retry usually follows a Create made moments earlier; a
cache that does not show it yet turned the refetch into NotFound and
failed a backup that had succeeded. Read it from the API server instead.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 3b86f57803 fix(backups): refuse to probe an S3 endpoint behind a path prefix
An endpoint such as https://s3.example.org/prefix cannot be addressed
by minio-go, which has no path-prefix option, and the prefix cannot be
added below the client either, because the SigV4 signature covers the
request path. The CNPG probe was worse than failing: it kept only the
host of endpointURL and listed the bucket at a location barman never
writes to.

Reject such endpoints with an explicit error, in the CNPG probe and in
the shared client, so the size is left unset instead of measured
somewhere else.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov 038f35376a fix(backups): do not report a partial listing or re-probe on retry
minio-go ends a listing whose context is cancelled between pages by
closing the channel without an error item. When the probe timeout
expired between pages of a large CNPG base backup, the pages already
read were summed and reported as the measured size. Check the context
once the listing channel closes and leave the size unset instead.

The probe ran before the Backup was created, so a reconcile that retried
after the Backup already existed probed the bucket again, held the
single BackupJob worker for up to the timeout, and then discarded the
result in favour of the existing object. Return an existing Backup
before probing.

Log the ambiguous MariaDB case (several dumps in the window) at info like
the other reasons the size stays unset, and cover the session-token
credential path with a test.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov f222e52863 fix(backups): harden the artifact size probe
The probe runs inside the reconcile that creates the Backup, and the
BackupJob controller has a single worker, so its timeout delays every
other BackupJob while an endpoint hangs. Lower it from 30s to 10s.

List the MariaDB prefix non-recursively, as mariadb-operator does. The
default MariaDB prefix and the default CNPG destinationPath are the same
<namespace>/<name>/ path, so a recursive listing of a MariaDB backed up
next to a same-named Postgres walked the whole WAL archive and ran out of
time. The CNPG base-backup prefix is still listed recursively.

Build the custom-CA transport from minio.DefaultTransport so it keeps the
proxy-from-environment and dial/TLS timeout settings a bare
http.Transport drops, and cover it with a test against a TLS endpoint
with a self-signed certificate.

Log the reasons the size could not be measured (credentials, client,
listing, empty CNPG prefix) at info. The chart runs the controller at
info, which hid them before. The expected ambiguity of several MariaDB
dumps in the window stays at debug.

Skip an endpointCA without a Secret name, as buildBarmanObjectStore does.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov de60488a2c fix(backups): report sizeBytes for CNPG backup artifacts
CNPG backups arrived without Backup.status.artifact.sizeBytes:
cnpg.io/Backup status carries no size, and the data lives in barman's
object-store layout rather than in a single object.

barman-cloud uploads every base backup under
<destinationPath>/<serverName>/base/<backupID>/ (backup.info plus the
data and tablespace tarballs, possibly split into parts). On completion
the driver now lists that prefix with the credentials and endpoint CA of
the rendered barmanObjectStore and records the total. The WAL archive is
not counted: it is one stream shared by every base backup of the server.
The field stays unset when the backup ID is unknown, the bucket cannot be
read, or nothing is stored under the prefix.

The S3 client and Secret lookup are now shared with the MariaDB driver.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Andrey Kolkov d4ad5612d3 fix(backups): report sizeBytes for MariaDB backup artifacts
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy,
so MariaDB backups always arrived without a size.

mariadb-operator writes each logical dump as
<prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by
every backup of the application, and the URI the driver records is
synthetic, so there is no single object to HEAD. On completion the driver
now lists the prefix with the tenant credentials the operator Backup was
rendered with and takes the size of the one dump whose timestamp falls
inside the operator Backup's lifetime. When the bucket cannot be read or
the window does not match exactly one dump, the field stays unset.

The API doc now states that an absent sizeBytes means "unknown", and the
Velero strategy documents why it leaves the field unset: kopia stores
data content-addressed across backups, so there is no per-backup stored
size.

Refs #4316

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-04 13:04:22 +00:00
Myasnikov Daniil 47b497194e fix(backups): restore the ClickHouse archive a Backup was taken as
The Altinity strategy named each archive inside its Pod with date -u
and recorded nothing on the Backup, so a restore could only guess: it
listed the remote archives under the application's prefix and took the
newest. A RestoreJob of an older Backup therefore restored whichever
backup had run last.

The controller now names the archive before it creates the backup Job,
from the BackupJob's persisted StartedAt, so every reconcile and every
retried Pod of the Job writes the same one. The name is recorded in the
Backup's driverMetadata next to the strategy parameters, and a restore
renders it into the strategy Pod, which restores exactly that archive
and fails, naming it, when remote storage no longer holds it. The name
is recorded only when the Job's Pod was given it, so a Job created by
an earlier controller, or a strategy that names its own archives,
leaves the Backup without one instead of pointing it at an archive
that does not exist.

Because a retried Pod reuses the name, it waits on an attempt the
sidecar is still running instead of being refused with 423, and asks
for resume, so a local or partial remote backup an earlier attempt left
is completed rather than refused as existing.

Backups taken before names were recorded stay restorable: the Pod picks
the newest archive whose name is stamped no later than the Backup's
takenAt, which is set only after the backup Job finished, and logs a
warning that the choice is a best effort.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 11:10:42 +00:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
IvanHunters a444ac4498 Merge branch 'main' into fix/tap-pin-digest
Reconcile the tap digest-pin change with #5071 (no-install-on-connect), now
merged into main. Keep this branch's resetTapPinState, the verify-before-pull
Trust text, and the index-README trust model, while adopting #5071's wording
that installing (not tapping) runs charts and dropping the stale
"re-registration" phrasing from the shared tap.go and rest.go comments.

Conflicts resolved in cmd/cozypkg/cmd/tap.go, pkg/registry/core/tap/rest.go,
and hack/packages-index/README.md.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-30 14:02:24 +03:00
Andrey Kolkov 07301c562c fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names (#5083)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #5014.

A bucket's `Workload` was named `"bucket-" + <BucketClaim name>`, and a
Bucket application names its claim after its release, which already
starts with `bucket-`. A bucket an operator knows as `cozy-backups`
therefore appeared as `bucket-bucket-cozy-backups`, and that string is
the `workload` of the bucket's consumer in every billing usage report.

- **Naming.** The `bucket-` prefix is now added only when the claim name
does not already carry it. A Bucket application's claim
`bucket-cozy-backups` gives the `Workload` `bucket-cozy-backups`; a
claim without the prefix, such as `my-bucket`, still gives
`bucket-my-bucket`. In the tree today every monitored BucketClaim comes
from the Bucket application (Harbor's registry claim carries no labels
its monitors select, so it has no `Workload` on either side of this
change).
- **Migration of existing Workloads.** Nothing garbage-collects bucket
`Workload`s by name: they are owned by their BucketClaim and live as
long as it does. Without a migration the old `bucket-bucket-…` object
would stay next to the new one and the bucket would be metered twice.
When the new `Workload` is reconciled, the controller looks up the
legacy name and, only if the same BucketClaim (matched by UID) is its
sole owner, uses its recorded sizes as the starting point and then
deletes it, with UID and resourceVersion preconditions so a `Workload`
that took over the name in the meantime survives. If the legacy
`Workload` has other owners as well, it is another bucket's renamed
`Workload` that a rolled-back controller attached this claim to: the
controller takes neither its sizes nor the object and only drops its own
owner reference. A legacy `Workload` with no owners left (a claim
deleted with `--cascade=orphan` and recreated) that still carries this
monitor's label is deleted the same way, without taking its sizes. Sizes
already recorded at the new name win over the legacy ones only when the
claim already owned that `Workload`. Carrying the sizes over matters
because the controller deliberately keeps the last known size when the
object-store metrics are unavailable; a fresh `Workload` would otherwise
report no size until the next successful poll, which billing reads as an
empty bucket.
- **Never taking over another claim's Workload.** With Bucket
applications `x` and `bucket-x` in one namespace, the legacy name of the
first (`bucket-bucket-x`) is the new name of the second. The controller
never takes over a `Workload` that has owners when the claim is not one
of them (in practice another BucketClaim). While the new name is held
that way, it keeps metering the bucket under its legacy name, without
migrating, and emits a `BucketWorkloadNameTaken` warning on the monitor.
For Bucket applications the other claim renames its `Workload` away on
its own reconcile and the migration completes on the next pass. Only
when the legacy name is held as well — which for a claim without the
prefix, whose two names coincide, means a claim `x` outside the Bucket
application next to the application `x` — is the bucket skipped, and the
warning says it is not metered. Without this, whichever claim reconciled
first decided whether the second bucket ended up with the first one's
size or with no size at all. The same rule keeps two claims that map to
one name (`x` and `bucket-x`) from sharing a `Workload`.

**Rollback.** The migration is one-way. A controller rolled back to a
previous version recreates `bucket-bucket-<name>` next to the renamed
`bucket-<name>`, which its monitor keeps alive, so the bucket is metered
twice until the `bucket-<name>` Workloads of Bucket applications are
deleted by hand. Upgrading again after such a rollback converges without
deleting another bucket's `Workload` (see above).

**Known limitation.** Taking over an ownerless `Workload` keeps the
sizes recorded on it as the last known ones, as the controller always
has. If such an object was left behind by another bucket (for example
`kubectl delete bucketclaim --cascade=orphan`), its sizes count for the
new claim until the next successful size poll.

**Effect on billing, worth knowing before the upgrade:** `Workload`
names are the identity billing integrates over. A bucket's consumer
changes name once, at the upgrade, so a usage report whose window spans
the upgrade lists the bucket twice: under the old name up to the switch
and under the new one after it. The totals are right; the split is
cosmetic. Reports for windows entirely before or after the upgrade are
unaffected.

Tests (`internal/controller/workloadmonitor_controller_test.go`):

- `TestBucketWorkloadName`: the naming rule for a Bucket application
claim, a plain claim and a Harbor claim.
- `TestReconcileBucketApp_NoDoublePrefix`: a Bucket application gets
`bucket-cozy-backups` and no `bucket-bucket-cozy-backups`.
- `TestReconcileBucketApp_ReplacesLegacyWorkload`: an existing
`bucket-bucket-cozy-backups` owned by the claim is deleted, and its
recorded logical and physical sizes (no metrics endpoint in the test)
are carried over to the new `Workload`.
- `TestReconcileBucketApp_KeepsLegacyNameOwnedByAnotherClaim`: a
`Workload` under the legacy name owned by a different claim is left
alone and its size is not taken.
- `TestReconcileBucketApps_LegacyNameIsAnotherAppsNewName`: Bucket
applications `x` and `bucket-x` with their legacy `Workload`s,
reconciled in both orders, end with `bucket-x` and `bucket-bucket-x`,
each with one owner and its own size, and the legacy
`bucket-bucket-bucket-x` gone.
- `TestReconcileBucketClaim_DoesNotAdoptAnotherClaimsWorkload`: a claim
whose name is held by another claim's `Workload` leaves it untouched,
emits `BucketWorkloadNameTaken` and is metered under its legacy name.
-
`TestReconcileBucketClaim_KeepsMeteringUnderLegacyNameWhileNewNameIsHeld`:
the legacy `Workload` keeps being updated, with its size and owner,
across repeated reconciles while the new name is held.
- `TestReconcileBucketClaim_UnprefixedClaimWithHeldNameIsSkipped` and
`TestReconcileBucketClaim_BothNamesHeldIsSkipped`: when no free name is
left, nothing is taken over and the event says the bucket is not
metered.
- `TestReconcileBucketClaim_UnprefixedClaimKeepsItsWorkload`: for a
claim without the prefix, where the legacy and the new name coincide,
its own `Workload` and size survive repeated reconciles.
- `TestReconcileBucketApp_RecordedSizeWinsOverLegacy`: a size already
recorded on the new `Workload` takes precedence over the legacy one.
- `TestReconcileBucketClaim_UpdatesItsOwnWorkload`: a claim's own
`Workload`, with its owner reference carrying the `BucketClaim` kind as
a live cluster's cache records it, keeps being updated and raises no
warning (the fake client leaves that kind empty on objects the
controller creates, so the other tests do not exercise this).
- `TestReconcileBucketClaim_AdoptsOwnerlessWorkload`: a `Workload` that
lost all owner references is taken over by the claim whose name it
carries, rather than blocking that name forever.
- `TestReconcileBucketApp_ReleasesSharedLegacyWorkload`: a legacy
`Workload` that another claim also owns (the rollback case) keeps its
size and its other owner; only this claim's reference is dropped, and
its size is not carried over.
- `TestReconcileBucketApp_KeepsWorkloadThatReplacedLegacy`: a `Workload`
that replaces the legacy one under the same name between the lookup and
the delete survives.
- `TestReconcileBucketApp_RemovesOrphanedLegacyWorkload`,
`TestReconcileBucketApp_KeepsOrphanOfAnotherMonitor`: an ownerless
legacy `Workload` with this monitor's label is removed without carrying
its sizes; one labelled for another monitor is left alone.
- `TestReconcileBucketApp_LegacySizeWinsOverOwnerlessNewWorkload`: a
stale size on an ownerless `Workload` at the new name loses to the
migrating legacy `Workload`'s size.
- `TestReconcileBucketApp_LegacyAlreadyGone`: a legacy `Workload` that
disappears between the lookup and the delete (NotFound) does not fail
the per-claim reconcile, and the size is still carried over.

Reverting the naming rule, the legacy delete, the size carry-over, the
owner check, the refusal to adopt, the same-name guard, the size
precedence ignoring NotFound on the legacy delete, the own-claim match
in the adoption check, the ownerless case, the physical size carry-over,
the sole-owner rule for legacy Workloads, the delete preconditions, the
legacy-name fallback or its own held check, the ownerless-legacy rule or
the size precedence on takeover each fails one of these. `go test
./internal/... -count=1` and `go vet ./internal/controller/` pass.

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, which touches only
`internal/controller/workloadmonitor_controller.go` and its test: no
package, values, schema, CRD, variant, `hack/` tooling, telemetry metric
or cozy-proxy contract changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(workloadmonitor): Workloads of Bucket applications are no longer named with a doubled "bucket-" prefix (bucket-bucket-<name> becomes bucket-<name>). Existing Workloads are replaced on upgrade, keeping their recorded size; billing usage reports whose window spans the upgrade list such a bucket once under each name. Rolling the controller back afterwards requires deleting the bucket-<name> Workloads of Bucket applications, or the buckets are metered twice.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Bucket workloads now use a consistent name without duplicating the
`bucket-` prefix.
* Existing workload size data is retained when workloads are renamed or
updated.
* Workloads owned by another claim are preserved. When a naming conflict
occurs, metering continues under the legacy name when possible; if both
names are unavailable, metering is skipped and a warning is emitted.
* Legacy workloads are cleaned up or updated according to their
ownership, helping prevent duplicate or orphaned workloads.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 13:43:42 +04:00
myasnikovdaniil 372c85575f fix(controller): carry healthCheckExprs and waitStrategy to existing releases (#5087)
## What this PR does

This PR syncs `healthCheckExprs` and `waitStrategy` from
ApplicationDefinition to HelmReleases that already exist. cozystack-api
writes both fields only when it builds a release, and the definition
reconciler copied chartRef, valuesFrom, disable-wait setting and labels
but not these two, so release created before its definition got
expressions keeps waiting on objects the definition already exempts.

Most visible case is vm-disk. Every disk created before the DataVolume
`current: "true"` expression was added still waits on its DataVolume, so
after platform upgrade an existing disk with import still in progress
fails the upgrade after 10 minutes with `timeout waiting for:
[DataVolume/<namespace>/vm-disk-<name> status: 'InProgress']`, then
retries forever and holds one of 5 workers of its helm-controller shard.

Reconciler now derives both fields same way the API does
(`config.ResolveWaitStrategy`) and syncs them in both directions, so
existing release ends up same as freshly written one. On current
definitions only vm-disk (DataVolume) and postgres (KEDA ScaledObject)
set expressions, both `current: "true"`, so this can only relax waits.
Releases of other kinds have nothing to sync and are not rewritten.

HelmReleases rendered by another chart (tenant modules like Etcd,
Ingress, Monitoring) carry Flux origin label
`helm.toolkit.fluxcd.io/name` and are skipped, their gate comes from
their own chart. Their chartRef is still synced as before.

Failed update was logged and dropped before, and nothing requeues the
definition for hours after that. Now errors are returned, so a conflict
is retried.

Unit tests cover both sync directions, `waitStrategy: legacy` without
expressions, no rewrite of in-sync release (nil vs empty slice too),
chart-rendered and other-kind releases left alone, and update failure
returned as error.

Fixes #4409

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(controller): existing HelmReleases now get `healthCheckExprs` and `waitStrategy` from their ApplicationDefinition. vm-disk and postgres releases created before these expressions were added stop waiting on DataVolume and KEDA ScaledObject during upgrades.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Helm releases now synchronize health-check expressions and wait
strategies configured in their application definition, including when
those values are removed or no expressions are specified.
* Releases with chart-rendered health-check settings retain those
settings, and releases outside the matching application kind are
unaffected.
* Update failures are now reported during reconciliation, while updates
to other releases continue. Reconciliation can retry after a failed
update.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-27 11:47:21 +05:00
myasnikovdaniil 7dc5c0b5e6 fix(marketplace): do not install a connected repository's components on connect (#5071)
## What this PR does

Connecting a marketplace repository materialized its `PackageSource`s
and then auto-created a tap-managed `Package` per source. The Package
reconciler installs a `HelmRelease` for every install-marked component
of the default variant, and Flux runs those under its `flux` service
account, which is bound to `cluster-admin` with no impersonation
(`--default-service-account` is not set, and neither `internal/operator`
nor `pkg/registry/apps` set `serviceAccountName` on the HelmRelease). A
publisher's install chart therefore ran as `cluster-admin` on connect,
with no operator step, contradicting the documented behaviour that
tapping installs nothing. The `privileged` flag is a prompt to the admin
and does not protect against a publisher acting in bad faith.

This removes the on-connect auto-registration (`ensureRegistration`).
Connecting now only materializes `PackageSource`s. Installing an app is
an explicit `cozypkg add`, which becomes the sole creator of a
`Package`; every other `Package` reference in the operator is a read or
a watch. Removing the call also fixes the second effect of the same
path: a `Package` the admin removed with `cozypkg del` is no longer
re-created on the next source revision.

A second commit reconciles the texts this behaviour change made wrong:
they promised that tapping registers a repository's apps in the catalog
and runs its charts. Reworded the `cozypkg tap`/`untap` help, the tap
success output (it now prints a `cozypkg add <name>` line per
materialized PackageSource), the `cozypkg init` scaffold README, the
packages-index trust model, and the dashboard Repositories banner and
per-repository empty state.

Accepted consequence: a connected repository's apps appear in the
catalog only after an admin runs `cozypkg add`, since the dashboard
catalog is computed from `ApplicationDefinition`s. Install from a
connected repository is CLI-only for now; the console connects and
disconnects repositories but does not create a `Package`. Populating the
catalog from `PackageSource` metadata without an install is the
long-term shape in #4359 and is out of scope here. The
privileged-confirmation gaps #4358 and #4359 stay open.

Upgrade note: a `Package` that Cozystack 1.7.0 auto-created on connect
stays after the upgrade, and the privileged-confirmation gate still
applies to it, so this is not a regression for existing installs. For a
newly connected repository no tap-labelled `Package` is created, so the
gate does not come into play on that path. An operator who does not want
a 1.7.0 auto-created package can remove it with `cozypkg del`.

Addresses #4978.

### Screenshots

Dashboard text-only change (Repositories banner and per-repository empty
state). No layout change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The diff is operator logic plus CLI/dashboard/doc text; it matches
no concrete trigger (no package, API, schema, tooling, annotation,
metric or commit-convention change). The dashboard console is vendored
in this repo (`packages/system/dashboard`), so its text change travels
in this PR. Leaving every box empty rather than ticking "No downstream
repository is affected".

### Release note

```release-note
fix(marketplace): connecting a marketplace repository no longer installs its components. A connected repository's apps are materialized as PackageSources but not installed until an explicit `cozypkg add`; install from a connected repository is CLI-only for now. Previously, connecting auto-created a Package whose install-marked components were deployed by Flux under its cluster-admin service account. A Package that Cozystack 1.7.0 auto-created on connect stays after upgrade and can be removed with `cozypkg del`.
```
2026-10-26 21:10:45 +05:00
IvanHunters 09b0688abd fix(marketplace): fail closed on an unparseable verified digest, and cover the generation gate
A verified cosign signature whose manifest digest could not be parsed fell back
to an unpinned tag pull, which reopens the tag-mutation window the verification
was meant to close. Return an error instead.

Add a test that isolates the status.observedGeneration == metadata.generation
conjunct of artifactReady; the existing tests exercised only the tag-prefix
conjunct, so a mutation of the generation check went uncaught.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 17:11:52 +03:00
IvanHunters 7494d1fad2 fix(marketplace): pin a tap digest only from a current-generation artifact
The materializer pinned whatever digest status.artifact reported, even when the
artifact still described the previous reference. On a re-connect to a new tag,
source-controller has not re-resolved yet, so the operator materialized and
pinned the old tag's content (issue #4979). Gate materialization and pinning on
the artifact reflecting the current spec: status.observedGeneration equals
metadata.generation, and for a tag-only source a revision that belongs to that
tag (<tag>@<digest>).

A tap connected before digest pinning existed already has its
materialized-revision equal to the artifact revision, so Reconcile short-circuited
and never pinned, letting it follow the tag one more time. Pin the current digest
before that early return.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 16:55:40 +03:00
IvanHunters 2f39e304f0 fix(marketplace): reconcile tap texts with no-install-on-connect
Removing on-connect auto-registration made several user-facing and doc texts
wrong: they promised that tapping registers a repository's apps in the catalog
and runs its charts. Tapping now installs nothing, so the texts told operators to
wait for apps that never appear.

Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a
`cozypkg add <name>` line per materialized PackageSource), the `cozypkg init`
scaffold README, the packages-index trust model, the dashboard Repositories
banner and per-repository empty state, and the stale re-registration code
comments. No behaviour changes beyond the added `cozypkg add` hint in the tap
output.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 15:18:14 +03:00
Myasnikov Daniil 8d8179f20a fix(controller): carry healthCheckExprs and waitStrategy to existing releases
cozystack-api writes healthCheckExprs and the resolved wait strategy
from the ApplicationDefinition only when it builds a HelmRelease, and
the definition reconciler that keeps existing releases in step copied
neither. A release created before its definition exempted an object
kept waiting on that object at every upgrade. For vm-disk that is
every disk created before the DataVolume expression was added: a disk
whose import is still running after a platform upgrade fails the
upgrade at the timeout and, under RetryOnFailure, retries forever
while holding a helm-controller worker.

The reconciler now derives both fields the way the API does, in both
directions, so an existing release ends up with what a fresh write
would produce. On the shipped definitions this only relaxes waits: the
vm-disk DataVolume and the postgres KEDA ScaledObject. A release that
another chart renders carries Flux's origin label and keeps the gate
its chart sets.

A failed update used to be logged and dropped, and nothing else
re-enqueues the definition for hours, so that release stayed out of
sync. The errors are now returned and the definition is retried.

Fixes https://github.com/cozystack/cozystack/issues/4409

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 14:35:34 +00:00
Andrey Kolkov a6c14f6834 fix(workloadmonitor): migrate ownerless legacy bucket Workloads and prefer legacy sizes on takeover
A BucketClaim deleted with --cascade=orphan and recreated leaves its
bucket-bucket-<name> Workload without owners. The migration recognised a
legacy Workload only by the claim's UID among its owners, so that object
stayed next to the new one with a frozen size and the bucket was metered
twice. Treat an ownerless legacy Workload that carries the monitor's
label as the claim's: delete it with the same preconditions, but take no
sizes from it, since they may predate the claim.

The size carry-over kept whatever the Workload at the new name had
recorded. When that Workload was not yet owned by the claim, an ownerless
leftover, its sizes were not written for this bucket, yet they won over
the migrating legacy Workload's. Prefer the recorded sizes only when the
claim already owned the Workload.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 17:45:04 +04:00
Andrey Kolkov 36ce21db79 fix(workloadmonitor): keep metering a bucket under its legacy name while the new one is taken
When the new Workload name was held by another owner the claim was
skipped altogether, so its legacy Workload stopped being refreshed and
billing kept reading the last size it had recorded. That happens briefly
with Bucket applications "x" and "bucket-x" until the first one has
migrated, and for good with a claim "x" outside the Bucket application
next to the application "x", which map to the same name.

Keep reconciling the bucket under its legacy name instead, without
migrating, until the new name is free. Skip it only when the legacy name
is held by another owner as well, which is the only option left for a
claim without the prefix, whose two names coincide; the warning event
then says that the bucket is not metered.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 17:34:25 +04:00
Andrey Kolkov fd69a8fbd6 fix(workloadmonitor): release, not delete, a legacy bucket Workload shared with another claim
A controller rolled back to the doubled prefix runs CreateOrUpdate on
bucket-bucket-x, which by then is the renamed Workload of the Bucket
application "bucket-x", and appends claim bucket-x as a second owner.
After the next upgrade claim bucket-x took that object for its own
legacy Workload: it carried the other bucket's size over and deleted the
object, and the other bucket's Workload came back without a size.

Treat a legacy Workload as the claim's own only when the claim is its
sole owner. When it has other owners, take neither its sizes nor the
object: drop the claim's owner reference and leave the rest.

Delete the legacy Workload with UID and resourceVersion preconditions,
so a Workload that took over the name between the lookup and the delete
survives; a conflict is not an error and the next reconcile retries.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 17:06:57 +04:00
Andrey Kolkov f9b7e936d7 refactor(workloadmonitor): leave alone any bucket Workload the claim does not own
The adoption check only refused a Workload owned by another BucketClaim,
and the kind test in it had no case that could tell it apart: a bucket
Workload only ever has BucketClaim owners. Refuse any Workload that has
owners when the claim is not among them, which needs no kind at all, and
keep taking over one that has lost every owner reference, since nothing
else would ever release its name. Pin the ownerless case.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 16:55:32 +04:00
Andrey Kolkov 54a916e31f test(workloadmonitor): pin that a claim keeps updating its own bucket Workload
On a live cluster the cache records the BucketClaim kind in a Workload's
owner reference, so a claim's own Workload passes through the "owned by
another claim" check on every reconcile, and only the own-UID match keeps
it from being skipped. The fake client leaves that kind empty, so no test
exercised it. Seed a Workload whose owner reference carries the kind and
the claim's UID, and expect it to be updated with no warning event.

Pin the physical size carry-over alongside the logical one, and word the
warning event so it holds when two claims map to one name for good.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 16:50:39 +04:00
myasnikovdaniil a26c441d78 fix(vm-disk): let CDI adopt the PVC a Velero restore recreates (#4955)
## What this PR does

After a Velero restore of a VMDisk or of a VMInstance (`RestoreJob` with
the Velero strategy), the disk's HelmRelease fails on its next upgrade
and stays failed:

```
Helm upgrade failed for release <ns>/vm-disk-<name> with chart vm-disk@...: failed to create resource:
admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request: Destination PVC <ns>/vm-disk-<name> already exists
```

The VMInstance release that references the disk then fails in turn
(`Specified disk not exists in cluster`). Seen on a v1.6.5 cluster with
CDI v1.66.1, where each failed upgrade is retried and the release
history of one such disk had grown past 46,000 revisions.

### Cause

The restore controller deletes the disk's DataVolume
(`prepareForRestore`) and Velero restores the PVC under the same name,
with no owner. The restored release matches its chart and values, so
Flux does not upgrade it and the DataVolume is not recreated right away;
the next Helm upgrade (a platform update, an edit of the VMDisk) does
recreate it, and CDI's webhook refuses it.

In CDI v1.66.1 the webhook admits a DataVolume over an existing claim
only when `ClaimMayExistBeforeDataVolume` is true
(`pkg/controller/common/util.go`): either the PVC carries
`cdi.kubevirt.io/storage.populatedFor: <dv name>`, or
`AllowClaimAdoption` returns true, which reads
`cdi.kubevirt.io/allowClaimAdoption` **from the DataVolume** and
otherwise falls back to the `DataVolumeClaimAdoption` feature gate. The
Velero strategy's resource modifiers
(`createResourceModifiersConfigMap`) put `allowClaimAdoption=true` on
the restored **PVC**, where CDI never reads it, and the `CDI` CR in
`packages/system/kubevirt-cdi` enables neither the gate nor anything
equivalent. Migration 29 creates its PVCs with the same PVC-side
annotation.

### Fix

The vm-disk chart sets `cdi.kubevirt.io/allowClaimAdoption: "true"` on
the DataVolume it renders, on the creation path and on the
existing-DataVolume path alike, so the annotation does not flap between
upgrades.

- A fresh disk is unaffected: no PVC exists when the DataVolume is
admitted, and the PVC CDI then creates carries
`cdi.kubevirt.io/createdForDataVolume: <dv uid>`, for which
`AllowClaimAdoption` returns false, so CDI imports, clones or uploads as
before.
- An existing DataVolume already controls its PVC, so the annotation
changes nothing there; a Helm upgrade only adds it to the DataVolume's
metadata, which CDI allows (only the spec is immutable).
- After a restore, the webhook admits the recreated DataVolume and the
DataVolume controller adopts the claim: `pvcRequiresWork` is false, it
takes controller ownership of the PVC and the DataVolume goes
`Succeeded` without populating it again, so the restored data is kept.

Checked against a v1.6.5 cluster carrying the broken state, by rendering
the chart for an affected disk and running `kubectl create
--dry-run=server`: the DataVolume without the annotation is refused with
the error above, the same one with the annotation is admitted. On that
cluster two affected disks were later recovered through the
claim-adoption path (the same `AllowClaimAdoption` check): the PVC kept
its uid, became controlled by the new DataVolume, and the DataVolume
reached `Succeeded` with no import.

### Alternatives considered

- **Enable the `DataVolumeClaimAdoption` feature gate in the `CDI` CR.**
It works too, but for every DataVolume in the cluster (tenant Kubernetes
worker disks, golden images, KubeVirt `dataVolumeTemplates`), each of
which would then silently take over any unowned PVC of its name. The
per-DataVolume annotation keeps the behaviour to the one chart whose
DataVolumes are expected to meet a restored claim, and CDI has said it
means to retire the gate (kubevirt/containerized-data-importer#3480)
while the annotation is checked first and independently of it.
- **Stamp `cdi.kubevirt.io/storage.populatedFor` on the restored PVC**
from the resource modifiers. That would also pass the webhook, but only
for disks restored by the Velero strategy, not for claims created by
migration 29 or by hand, and it would need a controller image change
where a chart change suffices.

The PVC-side `allowClaimAdoption` rule in
`createResourceModifiersConfigMap` (and the one in migration 29) is
inert under CDI v1.66.1. This PR leaves the rule and its unit-test
assertions in place, and corrects the comments that credited it with the
adoption (the collector and `createResourceModifiersConfigMap` in
`velerostrategy_controller.go`, and step 5 of
`examples/backups/vmi/scenario-user-restore.md`): they now say CDI
adopts the restored PVC through the annotation the vm-disk chart sets on
the DataVolume. The backports carry only the template and the unittest
suite, so these comment and doc edits do not reach them.

One behaviour change to be aware of: a PVC named `vm-disk-<name>` left
behind without an owner (a restored disk deleted before this fix, whose
PVC Velero restored without an owner) is now adopted by a new VMDisk of
the same name, with its old contents, instead of blocking that VMDisk
forever. That PVC lives in the same tenant namespace.

### Tests

- `packages/apps/vm-disk/tests/dv_claim_adoption_test.yaml`: the
post-restore state (a restored PVC, no DataVolume) renders the
DataVolume through the creation path with the annotation, and the
existing-DataVolume path keeps it. Both fail with the template change
reverted. `helm unittest packages/apps/vm-disk`: 116 passed.
- `examples/backups/vmdisk/05-restore-in-place.sh` (run by the
`vmdisk-2-backup-roundtrip` Chainsaw suite): after the restore, the
HelmRelease was already reported Ready because no upgrade ran, which is
why CI did not catch this. The script now forces a Helm upgrade
(`reconcile.fluxcd.io/forceAt` + `requestedAt`), then requires the
release Ready, the DataVolume `Succeeded`, and the PVC to keep its
restored uid and be controlled by the DataVolume.

### Backports

Opened by hand as #4956 (`release-1.6`) and #4957 (`release-1.7`) rather
than through the `kind/backport` labels: `examples/backups/vmdisk/` does
not exist on either branch, so the bot's cherry-pick would conflict.
They carry the template change and the helm unittest suite.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: a vm-disk template change (no
`values.yaml`, schema, `README.md` or `Chart.yaml` change), a new helm
unittest file, and a backup walkthrough script. No app, field, default,
CRD, platform value or tooling changes, and no documented workaround on
the website is made obsolete (the only mention of `allowClaimAdoption`
there is in the archived v0 docs, about a different manual procedure).

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-disk): a VMDisk restored by Velero, alone or as part of a VMInstance, no longer fails its next upgrade with "Destination PVC ... already exists": the DataVolume now carries cdi.kubevirt.io/allowClaimAdoption, so CDI adopts the restored PVC and keeps its data.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved in-place VM disk restores so restored storage is correctly
adopted when the disk resource is recreated.
* The restore process now waits for the disk to become ready and reports
an error if it fails or the restored storage is not retained.
* Clarified the adoption behavior in restore documentation and ensured
recreated disks use the required adoption setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-23 17:39:50 +05:00
Andrey Kolkov 730135903f test(workloadmonitor): pin that a vanished legacy bucket Workload is not an error
The legacy Workload can disappear between the lookup and the delete, for
example through owner garbage collection, and the delete ignores
NotFound for that reason. Reconcile only logs per-claim errors, so check
reconcileBucketClaimForMonitor's own result with a delete that reports
NotFound.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 15:43:29 +04:00
Andrey Kolkov a51ad0ed54 fix(workloadmonitor): never adopt a bucket Workload owned by another claim
With Bucket applications "x" and "bucket-x" in one namespace, the legacy
name of the first, bucket-bucket-x, is the new name of the second. When
the second reconciled first, CreateOrUpdate took over the first one's
legacy Workload, which already carried bucket x's size, and appended a
second owner. The first claim then found its own UID among the owners,
treated the object as its legacy Workload and deleted it, and the
second bucket's Workload came back without a size.

Leave a Workload alone while it is owned by a different BucketClaim, log
it and emit a BucketWorkloadNameTaken warning on the monitor. In the
case above the other claim renames its Workload away on its own
reconcile, and the name is free on the next pass. The same refusal keeps
two claims that map to one name, such as "x" and "bucket-x", from
sharing one Workload.

Pin the edge cases the migration depends on: the reconcile order above
in both directions, a claim without the prefix keeping its own Workload,
and a recorded size taking precedence over the legacy one.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 15:06:36 +04:00
Andrey Kolkov 164b5ec859 fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names
A bucket's Workload was named "bucket-" + the BucketClaim name, and a
Bucket application names its claim after its release, which already
starts with "bucket-". A bucket an operator knows as cozy-backups
showed up as bucket-bucket-cozy-backups in every billing usage report.

Add the prefix only when the claim name does not carry it. Claims from
other charts, such as Harbor's <release>-registry, keep their names.

Existing Workloads under the doubled name are owned by their claim and
would otherwise live on next to the new one, metering the bucket twice.
When the replacement is reconciled, take the legacy Workload's recorded
sizes as the starting point, so the rename leaves no gap in the metered
size, and delete the legacy Workload. Only a Workload owned by the same
BucketClaim is touched.

Billing sees the renamed Workload as a new consumer: a usage report
spanning the upgrade lists the bucket once under each name.

Fixes #5014

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-23 14:50:10 +04:00
IvanHunters c837838cc2 fix(marketplace): pin a tag-following tap source to the resolved digest
A tapped repository followed a mutable tag: the OCIRepository carried only
spec.ref.tag with a 5m interval, so Flux re-pulled whatever the tag pointed
at, and a moved tag reached running tenant application instances within the
interval. Whoever controlled the tag controlled what ran in the cluster.

The materializer now pins the source to the digest Flux resolved for the
artifact it just verified: it reads the manifest digest from
status.artifact.revision (source-controller formats an OCI tag revision as
'<tag>@<digest>') and writes it to spec.ref.digest, which Flux resolves
ahead of the tag. The tag is kept informational. Moving the tag upstream no
longer changes what the cluster pulls; an explicit re-connect is required to
move the pin. A user-supplied explicit digest is never rewritten, and a
revision without a well-formed digest leaves the source following its tag
rather than writing a malformed reference.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-22 22:02:49 +03:00
IvanHunters 3fa534f3cf fix(marketplace): do not install a connected repository's components on connect
Connecting a marketplace repository materialized its PackageSources and then
auto-created a tap-managed Package per source. The Package reconciler installed
a HelmRelease for every install-marked component of the default variant, which
Flux runs under its cluster-admin service account with no impersonation. A
publisher's install chart therefore ran as cluster-admin on connect, with no
operator step, contradicting the documented behaviour that tapping installs
nothing.

Remove the on-connect auto-registration. Connecting now only materializes
PackageSources; installing an app is an explicit cozypkg add (or dashboard
action), which is the sole remaining creator of a Package. This also stops a
Package the admin removed with cozypkg del from being re-created on the next
source revision.

Catalog population from PackageSource metadata without an install is left to
the long-term design in #4359, and the privileged-confirmation gaps #4358 and
#4359 stay open.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-22 21:50:26 +03:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Mattia Eleuteri b5d7e43c49 docs(backups): say CDI adopts a restored PVC through the DataVolume
The restore controller and the user-restore example described the
allowClaimAdoption annotation its resource modifiers put on restored
PVCs as what lets CDI adopt them. CDI v1.66 reads that annotation only
on the DataVolume; on a PVC it reads nothing but createdForDataVolume.
Adoption rests on the vm-disk chart annotating the DataVolume Flux
recreates, and a reader trusting the PVC rule could rely on it for a
restore without such a DataVolume, or drop the DataVolume annotation as
redundant.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-21 12:36:22 +02:00
myasnikovdaniil 1f92bcd20e fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings (#4914)
## What this PR does

Follow-up to the flux-plunger uninstall fence: the points the last
review round raised and left for after the merge, plus four found by a
later independent review. None changes the fence's design; each closes a
gap in what it reports, how long it waits, or what its documentation
says.

**Pre-delete hook.**

- A release being deleted that only the fence still holds is no longer
unfenced in the same step that lifts its suspension. helm-controller
runs the uninstall of a release being deleted once it is no longer
suspended, but removing the fence at once deleted the object before
helm-controller could reconcile it. The hook now keeps that fence while
helm-controller handles the generation the lifted suspension wrote,
until shortly before its retry window closes, and names a release it
still had to let go of as before. A release whose state cannot be read
keeps its fence for the wait the same way, rather than being let go of
on one failed read.
- A release being deleted while still suspended, by anyone but
flux-plunger, was reported as cleanly released, although helm-controller
skips the uninstall of a suspended release. It is now named as deleted
without an uninstall and counted with the others.
- Each release gets its own share of the retry window, capped at the
window's end. One release that could never be read used to take the
whole window and leave every release listed after it a single attempt.
The share is a cap, not a floor of attempts: a floor would let enough
failing releases run the Job past its deadline, which kills it before
its report is written.
- The hook stops, waits for and lists again only for the flux-plunger of
its own release. Another one (a second installation under another
release name, changed labels, a binary outside the cluster) is not
stopped by the hook, keeps the fences it sets and gives them up itself;
the README now says so instead of the hook re-listing unconditionally,
which would not close that case and would fail the uninstall for as long
as a second installation is recovering anything.
- helm-controller's finalizer, which the hook reads to tell whether a
release being deleted is still held, is tied to the constant the
controller uses, as the fence already was. Two comments are corrected:
the retry window counts from the script's start, not the Job's, and
kubectl scale's internal conflict retry never runs here.

**Shutdown drain.**

- A drain deadline of zero or less is refused at startup, and the
chart's lowest accepted grace period is held to the same constants as
the drain subtraction, so raising the manager's stop bound cannot make
the chart render a deadline that is already spent. The timings the
README and the values comment state are held to those constants too.
- Fences left behind are reported as an error naming each release, not
an Info line; a drain that never got a listing answered says so instead
of reporting a count of -1; a flux-plunger that never led no longer
claims it released every fence.
- The drain's Events fold repeats into one counted Event per release and
reason, as the manager's recorder does, instead of one object per pass.
The fold is a patch, the verb the chart grants on events, and every
write is bounded by the drain's deadline. It does not add a per-release
Event for fences left at the end: by then the drain's deadline is spent,
and writing one would reach past it.
- Tests now hold the outcome line, the retry of a release whose fence
cannot be given up yet, and the pause between passes, none of which any
suite noticed when removed.

**Controller.** The `SuspensionNotOwned` and `TerminatingWhileSuspended`
Warnings were keyed on the processed-version annotation, which stays on
every release flux-plunger ever recovered, so a later restore, migration
or operator suspension of a healthy release drew a Warning telling the
reader to resume it, and a migration's deliberate suspend-then-delete
was called an accident. Both now key on the state a leftover actually
freezes the release in, `Ready=False` with `has no deployed releases`,
since helm-controller rewrites no condition of a suspended release; the
README's candidate query follows. Two comments that claimed more than
holds are qualified.

**Chart and README.** A `shutdownGracePeriodSeconds` of a million or
more now gets the range message rather than being refused as a fraction.
The README gives the recovery steps in the order the controller runs
them (the annotation is written before the Secret is deleted), and names
the window in which a backup restore started during a recovery is
unsuspended under it, since the etcd, CNPG and Velero restore steps
write `spec.suspend` only when it differs.

### Testing

Each change was written test first: a test that builds the binary and
runs it with a deadline of zero, helm-unittest for the chart, unit tests
with a captured logger and the fake client for the drain's outcome,
Events and passes (the Event fold under a client that refuses the update
verb, as the chart's role does), table tests for both Warnings, and the
bats suite for the hook, whose fake kubectl now models helm-controller
handling a release some reads after its suspension is lifted and a state
read that fails. Every fix was then reverted on its own, and the suite
went red each time. The envtest suites pass unchanged. The README's
candidate query was run against HelmReleases read from a live cluster,
with a stuck, a healthy, an owned and an unsuspended variant, and lists
only the stuck one.

### Screenshots

Not applicable: no UI change.

### Downstream repositories

The trigger map was walked against the diff:
`internal/controller/fluxplunger`, `cmd/flux-plunger`, the
`packages/system/flux-plunger` chart (templates, tests, README) and its
bats test under `hack/`. The website's FluxPlunger section in
`guides/concepts.md` draws the recovery with the Secret deleted before
the annotation is written, which is the order before the fence change;
it needs the same correction as the chart README. That change is held as
a draft until v1.7.0 is released and the website's `next/` is
re-initialized: the fence change is not part of release-1.7, so the
corrected order would be wrong for v1.7 docs, which are copied from
`next/`. No other repository is reached by this diff.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/752 (draft, merge
after v1.7.0)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(fluxcd): flux-plunger's pre-delete hook now lets helm-controller run the uninstall of a release being deleted before it removes the release's fence, gives each release its own share of its retries, and reports releases deleted under another manager's suspension; the shutdown drain reports the fences it leaves as errors and refuses a deadline that leaves no time
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Shutdown drain timeouts must be positive, and invalid values now stop
startup with a clear error.
* Uninstall cleanup waits for Helm to handle releases before removing
their fences, while bounding retries and reporting releases that remain
incomplete.
* Recovery and suspension warnings now reflect the release’s actual
failure state, and recovery can retry stale storage cleanup against the
same revision.

* **Documentation**
* Clarified shutdown grace-period requirements, uninstall behavior, and
guidance for backup restores and existing suspensions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-20 17:44:34 +05:00
Andrey Kolkov 9af1c0f902 fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so
prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and
no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the
restored HelmRelease reinstalled and CDI recreated the DataVolume over the same
PVC name the data mover was repopulating, wedging the RestoreJob.

Report a standalone VMDisk's own disk as its underlying resource, so the existing
(VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease,
deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before
the data mover recreates and repopulates it. No VM, so no IP/MAC is collected.

A VMInstance is halted before its disks are touched; a VMDisk target is not,
and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM
mounts leaves the original Terminating behind pvc-protection, Velero skips a
PVC that still exists, and the RestoreJob would report Succeeded having moved
nothing while the VM loses its disk on the next restart. An in-place VMDisk
restore now fails before changing anything when a Pod that has not terminated
mounts the disk's PVC, naming the Pod and telling the user to stop the VM or
restore the VMInstance instead.

A VMDisk Backup taken before the disk was recorded as its underlying
resource has none, so its disk is now derived from the application name
and such a restore is prepared and checked for a mounting Pod as well.

The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk
strategy already selects its disk, and Velero rejects a Backup or
Restore that carries both labelSelector and orLabelSelectors, so
appending them for a VMDisk would break a strategy written with
labelSelector.

When an in-place restore fails after preparation, nothing is rolled
back: resuming the disk's HelmRelease while its PVC is renamed and its
DataVolume deleted would have CDI import a fresh disk under the original
name, which looks healthy and hides that the data sits in the -orig-
PVC. The failure message and a Warning event now name the suspended
HelmReleases and the -orig- PVCs so the user can recover them, read back
from the cluster, so a failure before a step ran does not name what it
would have left.

Preparation takes the target apart before Velero validates the Restore,
so it is refused up front when the Velero Backup is gone or not
Completed. That check now runs on every pass behind the same failure
path as the rest of preparation, and a FailedValidation Restore fails
the RestoreJob like a Failed one instead of leaving it Running. A
HelmRelease suspend or a VM halt that did not happen now stops
preparation instead of renaming and deleting under a running release or
VM, and a VMInstance's disks are renamed only once its launcher Pod has
let go of them. A VMDisk held only by a Pod already being deleted is
waited for rather than refused, for
up to 30 minutes after the RestoreJob started.

Between renaming a kept disk's PVC and deleting its DataVolume, CDI can
recreate an empty PVC under the original name. Velero then skips the
PVC as already present, the RestoreJob reports Succeeded and the VM
boots a blank disk. Once the DataVolume is gone, preparation now
deletes a PVC found under the original name, read uncached, before the
Velero Restore is created.

Preparation spans several reconciles, so the RestoreJob records a
PreparationStarted condition before it first changes the target, and
retries without changing anything when recording it fails. A
refusal on a later pass (the Velero Backup expired, a VM started on the
disk) then names what preparation left behind instead of claiming that
nothing changed, and a preparation still waiting 30 minutes after it
started fails the same way instead of polling forever.

Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it
too: its DataVolume is deleted with orphan propagation and the bound PVC
is deleted before the Restore, as for a VMInstance disk. Left in place,
Velero would skip it and CDI would refuse the DataVolume the resumed
HelmRelease creates.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-13 18:05:45 +04:00
Alexey Artamonov 699555c7cb fix(fluxcd): give each release its own share of the hook's retries
The pre-delete hook's retry budget was one window spent in listing
order, so a release that could never be read or unfenced, listed
first, took all of it and every release after it got a single attempt.
The same transient failure then released a fence or not depending on
where its release sorted. Each release now gets its own share of the
window, capped at the window's end, and the steps that cover every
release keep the whole of it.

Two comments said more than holds. The retry window is measured from
the script's start, not the Job's, so scheduling and the image pull come
out of its one-minute reserve rather than being covered by it. And the
five-minute conflict retry inside kubectl scale never runs here, since
without a precondition it sends a merge patch with no resourceVersion
to conflict on.

helm-controller's finalizer, which the hook reads to tell whether a
release being deleted is still held, is now tied to the constant the
controller uses, as the fence already was.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Alexey Artamonov 7b84919db3 fix(fluxcd): warn about a foreign suspension only on a stuck release
flux-plunger raised its SuspensionNotOwned and TerminatingWhileSuspended
Warnings on a suspension it does not own when the release carried its
processed-version annotation. That annotation is never removed, so
every later suspension of such a release, by a migration, a backup
restore or an operator, drew a Warning: one telling the reader to
resume it, which mid-restore is the one thing nobody wants, and one
calling a migration's deliberate suspend-then-delete an accident. Both
also missed the leftover they are for when the first recovery stopped
before writing the annotation.

A suspension an earlier flux-plunger left mid-recovery freezes the
release in the failure recovery started from, since helm-controller
rewrites no condition of a suspended release. Both Warnings now key on
that state instead, and the README's candidate query with them.

Two comments are qualified where they claimed more than holds: a stop
that overran the manager's grace period can still let a reconcile set
a fence after the drain's last listing, and a process stopped before
controller-runtime closes Elected holds the Lease yet does not drain.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Alexey Artamonov caeaa6b5e5 fix(fluxcd): make flux-plunger's shutdown drain report what it left
The drain's only report was an Info line, so an alert on errors never
saw fences left behind; it named no release, and printed held: -1 when
no listing had answered. A flux-plunger that never led logged that it
had released every fence next to the line saying it drained nothing.
Fences left behind are now an error naming each release, an unanswered
listing says so, and a process that never led says only that.

The drain writes its Events itself, since the manager's recorder stops
with the manager, but it created one Event object per pass for a
release it revisits every second, and each write had a deadline of its
own outside the drain's. Repeats now fold into one counted Event, as the
manager's recorder folds them, and every write is bounded by the drain.

Tests now hold the outcome line, the retry of a release whose fence
cannot be given up yet, and the pause between passes, none of which any
suite noticed when removed.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Alexey Artamonov 8bcb709ce8 fix(fluxcd): refuse a flux-plunger drain deadline that leaves no time
A drain deadline of zero or less is already spent when the drain
starts, so flux-plunger would exit holding every uninstall fence
without listing one. The chart could produce one: its 40-second floor
was a literal, so raising the manager's stop bound or the margin before
SIGKILL, which the drain subtraction is already tied to, would leave the
floor rendering a non-positive deadline. The binary now refuses such a
flag at startup, before it reaches for a cluster, and the chart's floor
and the timings the README and the values comment state are held to
the same constants.

A whole number of a million seconds or more printed in exponent form
and was refused as a fraction; the range is now checked first, so it
gets the range message.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-13 15:09:00 +03:00
Andrey Kolkov 0c2f131192 feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.

The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.

The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.

The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-10-13 12:34:24 +04:00
Aleksei Sviridkin 785cbc391b fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (#4293)
## What this PR does

Fixes #4060.

`flux-plunger` recovers a HelmRelease stuck with `has no deployed
releases` by briefly suspending it, deleting the stale Helm storage
Secret, recording a processed-version annotation, and unsuspending.
helm-controller v1.5.x skips the Helm uninstall and drops its finalizer
when it handles a delete on a suspended HelmRelease, and it wakes on the
same delete event flux-plunger does, so a delete arriving inside that
window orphaned every resource the release owns (for a VMDisk:
DataVolume, PVCs, the CDI importer pod, RBAC, WorkloadMonitor and the
storage Secret itself). Lifting the suspension on the delete event
cannot win that race reliably. The bug is not specific to VMDisk.

The change makes such a delete reach the uninstall:

- **An uninstall fence.** The suspension is applied together with a
flux-plunger finalizer, `flux-plunger.cozystack.io/uninstall-fence`, in
one server-side apply, so flux-plunger never holds a suspension without
it. If a delete arrives while the release is suspended, the fence keeps
the object whether helm-controller or flux-plunger handles the delete
first: flux-plunger lifts its suspension and keeps the fence,
helm-controller reconciles the unsuspended generation and runs the
uninstall, and flux-plunger removes the fence once helm-controller's
finalizer is gone and it has observed that generation. A recovery that
completes lifts the suspension and the fence together. A terminating
HelmRelease never begins recovery, and termination is re-checked
immediately before deleting the storage Secret.
- **Only flux-plunger's own suspension is acted on.** Ownership is
decided by which field manager owns `spec.suspend` in
`metadata.managedFields`: the apiserver transfers the field away the
moment an operator (or `cozyhr resume`, or a backup/restore controller)
writes a different value to it, so a later suspension is never misread
as flux-plunger's. A suspension set through the shared
`flux-client-side-apply` manager (platform migrations, the tenant delete
hook, an operator's manual `kubectl apply`) is never claimed. A release
deleted under another manager's suspension is that manager's decision:
flux-plunger does not hold it with the fence, and helm-controller leaves
its resources in place, as the migrations that suspend and then delete
rely on.
- **No suspension taken over in a race.** Acquiring applies without
forcing first and, on a conflict, re-fetches and forces only if
`spec.suspend` is still `false`: a backup controller owns the field at
`false` via a dynamic-client `Update`, which a non-forced apply can
never take. Every apply flux-plunger sends, acquiring as well as lifting
the suspension or the fence, is preconditioned on the `resourceVersion`
and `uid` of the read that decided on it. The `resourceVersion` makes a
write landing in between fail the apply instead of being absorbed:
acquire then defers rather than adopt or take over a suspension set
meanwhile, and lifting the suspension and the fence from a release read
as live cannot delete it without an uninstall if helm-controller has
handled a delete in the meantime. The `uid` stops an apply to a release
that has gone from creating an empty one, since server-side apply
creates what does not exist. A conflict is retried through the requeue.
The apply carries only `spec.suspend` and the fence, so it cannot leak
the required `spec.interval` and conflict on it.
- **Fresh reads.** The release, every re-check and the storage Secret
list are read from the apiserver (an uncached `APIReader`), not the
informer cache. A lagging cache would let the processed-version guard
delete the revision below the one just reclaimed, and the ClusterRole
grants secrets `get`/`list` but not `watch`, which an informer needs to
stay synced; flux-plunger deliberately does not cache Secrets. Every
step requeues with backoff on failure.
- **Events.** A Warning when flux-plunger holds a suspension it failed
to clear, and a Warning when a delete has been held by the fence for
more than ten minutes waiting for helm-controller (the held state is
re-checked on a timer, not only on helm-controller's next status write).
A release deleted under another manager's suspension gets a Normal event
when that is a migration's choice, and a Warning when the release
carries flux-plunger's processed-version annotation, because then the
skipped uninstall is an accident, not a decision. Normal events also
mark an externally-owned suspension left alone and a recovery deferred
because the release changed between read and write; an external
suspension on a release flux-plunger has recovered before is a Warning
instead, because it is most likely one an earlier flux-plunger left
mid-recovery (see Operating notes). Events from the shutdown drain are
written straight to the apiserver, since the manager's recorder stops
with the manager.
- **The fence is released when flux-plunger goes.** Once its manager has
stopped, and with it every reconcile, flux-plunger drains its fences
under a deadline inside the pod's termination grace period; a stop that
overran its 10s bound is drained too and still reported: a recovery in
progress is aborted, the suspension and the fence lifted together for
the next flux-plunger to start again, and a release being deleted has
flux-plunger's suspension lifted and keeps the fence until
helm-controller has run its uninstall. A release helm-controller does
not handle in time keeps its fence. The drain lives in the controller,
so the envtest suites hold it against a real apiserver's generations and
finalizers. A pre-delete hook on the chart is the backstop for what the
drain could not finish, so uninstalling the component never leaves a
release it cannot finish deleting: it stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and fence without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, the outcome the controller's own relinquish has, with a JSON
patch of `spec.suspend` preconditioned on the `resourceVersion` and
`uid` it read; not with `kubectl apply --server-side`, which on a
release last written by a client-side `kubectl apply` moves that apply's
fields to the applying manager and strips every one the body leaves out.
helm-controller runs a pre-delete hook under the release's uninstall
timeout, 300s by default and not set by the operator, so the Job's
deadline sits under it. A Deployment that cannot be stopped or a listing
that never succeeds ends the hook before any fence is touched, and a
fence it cannot remove ends it after the sweep; either way it exits
non-zero, because a failed pre-delete hook leaves the release untouched
and helm-controller retries the uninstall, hook included, whereas
exiting 0 would let the component go and strand the fence. A
flux-plunger pod still there after its grace period, typically one stuck
`Terminating` on a `NotReady` node, is only reported, and the pod wait
is cut to what is left of the hook's retry window; because such a pod
may still be setting fences, the fences are listed again after the sweep
and any found fails the hook. The Job runs once and is kept after
success so its report stays readable until its TTL;
`helm.sh/hook-output-log-policy` is deliberately not used, because
helm-controller never sets Helm's hook output writer and calls a nil one
for it (fluxcd/helm-controller#1562). The hook's kubectl image is
digest-pinned in the chart values, and its RBAC is only what `kubectl
scale`, the listing and the patches issue. The chart's README documents
the finalizer and how to clear it by hand.

A single writer is kept by leader election, not by the rollout. A second
process acting under the same field manager could clear a suspension the
first is still recovering, and no Deployment strategy prevents one: the
next pod starts as soon as the old one is terminating (Kubernetes v1.35
has no pod-replacement policy for Deployments), and an eviction or a
node drain replaces a pod the same way, while the old one stops and
drains. The chart now passes `--leader-elect` with a Role on `leases` in
the release namespace. The Lease is not given up on stop and outlasts
the last renewal, the renew deadline, the manager's bounded stop (10s)
and the drain, and only a flux-plunger that led drains at all, so the
next one acts only once the old one is done; the price is that a new pod
waits for the Lease to run out, a little under two minutes at the
default. The timings derive from one value, `shutdownGracePeriodSeconds`
(90 by default, refused outside 40 to 140 at render): the pod's
termination grace period, the drain deadline (25s less: the manager's
10s stop plus a 15s margin before the kubelet's SIGKILL, both tied to
the Go constants by a contract test) and the hook's pod wait. The
strategy keeps `maxSurge: 0` and `maxUnavailable: 1` and stays
`RollingUpdate` rather than `Recreate` on purpose: the chart shipped
without a strategy block, so every existing install's live Deployment
carries the apiserver-defaulted `rollingUpdate` object and
helm-controller applies server-side, where laying `type: Recreate` over
that populated block is rejected on upgrade.

### Operating notes

A HelmRelease carries the fence only while flux-plunger is recovering it
or waiting for helm-controller to uninstall it. A fenced release whose
uninstall keeps failing stays `Terminating` exactly as any HelmRelease
does under helm-controller's own finalizer, and raises the Warning
above. Stopping flux-plunger drains its fences, and uninstalling it
releases whatever the drain left through the pre-delete hook; a fenced
release that is deleted while flux-plunger is merely scaled to zero, or
killed without a graceful stop, and that helm-controller has not
uninstalled, stays `Terminating` until flux-plunger runs again. To
release one by hand, after confirming its uninstall has run or is not
wanted:

```bash
kubectl patch helmrelease NAME --namespace NAMESPACE --type json --patch '[{"op":"test","path":"/metadata/finalizers/INDEX","value":"flux-plunger.cozystack.io/uninstall-fence"},{"op":"remove","path":"/metadata/finalizers/INDEX"}]'
```

where `INDEX` is the position of
`flux-plunger.cozystack.io/uninstall-fence` in `metadata.finalizers`;
the `test` operation makes the patch fail rather than remove another
finalizer if the list has changed.

Earlier flux-plunger releases set `spec.suspend` through the shared
`flux-client-side-apply` manager, so a suspension one of them left
behind cannot be told apart from a migration's or an operator's and is
treated as external: it is not cleared automatically. Such a leftover
exists only if the old pod stopped in the middle of a recovery, and
flux-plunger raises a `SuspensionNotOwned` Warning on it. The chart
README carries this section too. To list candidates, suspended releases
carrying the flux-plunger annotation whose `spec.suspend` is not owned
by flux-plunger's apply:

```bash
kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | jq --raw-output '.items[] | select(.spec.suspend == true) | select(.metadata.annotations["flux-plunger.cozystack.io/last-processed-version"] != null) | select([.metadata.managedFields[]? | select(.manager == "flux-plunger" and .operation == "Apply" and .fieldsV1["f:spec"]["f:suspend"] != null)] | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"'
```

Clearing one is an operator action: confirm nothing else (a migration, a
restore) is holding it, then resume it with `flux resume helmrelease
NAME --namespace NAMESPACE`.

### Testing

The regression the issue asks for is
`TestEnvtest_DeleteDuringRecoveryReachesUninstall`: against a real
apiserver, a delete lands while flux-plunger holds its recovery
suspension, and helm-controller's v1.5.0 delete handling is reproduced
step for step (uninstall only when not suspended, finalizer dropped
either way, observed generation recorded). It covers both orders,
helm-controller handling the delete first and flux-plunger handling it
first; without the fence the first order leaves the release gone with no
uninstall. The same suite pins the managedFields ownership dynamics a
fake client cannot represent: the field transfer, the forced acquire
over a dynamic-`Update` owner, the `resourceVersion` precondition
rejecting a write that races either apply, the boundary that a
`flux-client-side-apply` suspension is never claimed, and interval
preservation. The fence mechanics were also checked against a live
helm-controller: with another finalizer holding a deleted, suspended
release, lifting the suspension made helm-controller run the uninstall
and remove the release's resources.

The envtest suite runs on every pull request: a new `make
test-controllers-envtest` target builds a pinned `setup-envtest` and
fetches a kube-apiserver and etcd into `_out/envtest`, which CI caches
and the target reuses without asking the release index, takes the
HelmRelease CRD from the in-tree Flux manifests, and CI calls it next to
`make test-controllers`; the control plane gets a longer start timeout
because it starts beside the other test goals. The drain is held there
too: it aborts a live recovery, carries a delete through the uninstall
and lifts the fence only once helm-controller has observed the
unsuspended generation, keeps the fence when helm-controller never
answers, releases a fence left without a suspension at once, and leaves
another manager's suspension alone.
`TestEnvtest_TheNextFluxPlungerLeadsOnlyAfterTheDrain` runs two real
managers on one Lease: a clean stop still drains, and the second manager
becomes leader only after the first one's drain has ended; it goes red
if the Lease is given up on stop. Unit tests cover the controller logic
(every branch, ordering, requeue, and the ownership decision through an
injected seam), the Lease timing relations, that only a leader drains
and through the manager's API reader rather than its stopped cache, that
a failed listing is retried, that a stop which overran its grace period
is still drained and its events reach the apiserver, and contract tests
tie `main.go`, the chart's flags, the drain's arithmetic and the hook's
names to the Go code. The pre-delete hook is covered by helm-unittest
and by `hack/flux-plunger-fence-hook_test.bats`, which runs its rendered
script against a fake `kubectl` that keeps each release's finalizers,
suspension owner, generation and observed generation, bumps the
generation on a spec write as the apiserver does, answers NotFound
without `--ignore-not-found`, spends simulated time on waits, lists a
release once however many copies of the fence it carries, refuses
`kubectl apply`, lifts a suspension only through the preconditioned JSON
patch and never one another manager also holds, reads a negative
`kubectl wait` timeout as a week as kubectl does, and applies a
finalizer removal only when it is the guarded test-then-remove at an
index that holds the fence. Every mutation in a set covering each
predicate conjunct, each retry, each exit path, the re-listing, the
timings and the selector turns the hook suite red, as does every
mutation in the Go set covering the drain's branches, the Lease timings
and the events; three drain mutations that still survive are tracked in
#4782. The hook script was also run as a Job in the hook's own image on
a live cluster, succeeding, with the Deployment already gone, and with
patch denied by RBAC, where it exited non-zero and named the release it
left; that run caught that a per-call `--request-timeout` makes kubectl
drop its in-cluster config, so the hook sets none and the fake rejects
it. A later live run caught `kubectl apply --server-side` stripping the
finalizers and spec of a release created by a client-side apply; with
the JSON patch the same releases kept every field, the suspension
flux-plunger alone held was lifted, a co-owned and a foreign one stayed,
and every fence went.

### Remaining limitations

- A delete that arrives after recovery has deleted the storage Secret of
a release whose only revision was that one finds no revision to
uninstall; helm-controller then treats the release as already gone.
- A writer that sets `spec.suspend: true` while flux-plunger already
holds it at `true` does not take the field over, because the value does
not change. If such a writer then deletes the release inside the
recovery window (the suspend-then-delete that the tenant delete hook and
some platform migrations use to skip the uninstall), flux-plunger still
sees its own suspension, lifts it, and helm-controller runs the
uninstall that writer meant to skip. The same held before this change,
when the end of recovery unsuspended unconditionally.
- The already-processed guard compares revision numbers only. When Helm
re-creates a revision under the same number after recovery deleted it (a
release whose only revision fails again the same way), recovery runs
again for it.
- `listHelmReleaseSecrets` reads the local cluster only, so a
HelmRelease that targets a remote cluster via `spec.kubeConfig` (whose
storage Secrets live in the remote cluster) is not recovered; it is
skipped, never mis-handled.
- A new flux-plunger pod waits for the old one's Lease to run out before
it acts, on every restart and after a crash: a little under two minutes
at the default. Recovery is not latency-sensitive, and this is what
keeps a single writer while the old pod drains.
- The leader Lease is left in the release namespace after the chart is
uninstalled, as with any controller-runtime leader election; it is
inert.

### Screenshots

Not applicable: no UI change.

### Downstream repositories

The trigger map was walked against the diff: it touches
`internal/controller/fluxplunger`, `cmd/flux-plunger`, the
`packages/system/flux-plunger` chart (templates including RBAC for the
leader Lease, a pre-delete hook, values, tests, its package `Makefile`,
README), a new bats test under `hack/`, the root `Makefile` (a new
`test-controllers-envtest` target; existing targets are unchanged) and
the pull-request workflow that calls it and caches its binaries. No row
matches: ccp gates on existing `hack/` files and make targets, and
nothing there is moved, renamed or changed in behaviour; a new test file
and a new target reach none of them. The website's FluxPlunger section
describes the recovery steps, which still hold; it does not describe the
fence or the drain, which the chart README now documents.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(fluxcd): a HelmRelease deleted while flux-plunger is recovering it now reaches the Helm uninstall instead of leaving its resources (DataVolume, PVCs, RBAC, storage Secret) behind
```
2026-10-12 15:23:49 +03:00
Aleksei Sviridkin 8deaa8e14c fix(vm-disk): name the importer error in the disk's WorkloadsReady message (#4815)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A VMDisk whose HTTP source answers 404 or does not resolve never
finishes importing, and since #4380 its `WorkloadsReady` condition says
only `DataVolume vm-disk-x is ImportInProgress`. That reads the same as
a large download that is still running. CDI fails an import only on a
fatal error (a checksum mismatch) and restarts the importer pod
otherwise, so the phase never moves, and the reason the disk will not
finish was visible only in the importer logs.

CDI already keeps that reason on the DataVolume: the `Running` condition
goes `False` with the importer's termination message, and keeps it
through the back-off between restarts. The WorkloadMonitor now appends
it, so the condition reads `DataVolume vm-disk-x is ImportInProgress:
Unable to connect to http data source: expected status code 200, got
404. Status: 404 Not Found`. The console already shows the
`WorkloadsReady` message on the disk's overview page, so no UI change is
needed. The same text covers an unschedulable importer, a failed image
pull and a checksum failure, since CDI reports all of them through
`Running=False`.

Only `Running=False` is read: `Unknown` is not a failure, and `Bound`
repeats the same error behind PVC details. For a host-assisted clone CDI
joins the target and source pod messages as `<target> and <source>`,
which comes out as ` and ` when both pods are still being created, so
empty halves are dropped.

Behaviour to be aware of: while the importer pod is briefly running
between restarts, CDI flips `Running` to `True` and clears the message,
so the condition shows the bare phase for those seconds. This is not a
terminal state either: a disk with a broken source still stays
`ImportInProgress` and keeps retrying. Deciding when such an import
should count as failed is left for a separate change.

### Tests

A table test covers a 404, an unresolvable host, an unschedulable
importer, a fatal checksum error, a DataVolume without a phase and
joined clone messages with one or both halves empty, plus the cases that
must not add text: an importer that has not started, a running importer,
an `Unknown` condition, a populated disk and an error that only sits on
`Bound`. Each conjunct of the condition check and the append itself were
mutated and each mutant fails the test. A second test runs against a
DataVolume captured from a test cluster after its importer had restarted
about two thousand times on a 404, with only names and the URL changed,
so the parsing is pinned against what CDI actually writes rather than a
hand-built shape.

### Screenshots

No UI code changes: the console renders the `WorkloadsReady` message it
already receives. A message that carries a long source URL is not
wrapped on the overview card; that is a styling follow-up for the
console and is left out of this change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only non-Go change is a sentence in the
`packages/apps/vm-disk/README.md` notes; no values, schema, API kind or
default changes. The website reference page for vm-disk is regenerated
from that README by the release bot.

### Release note

```release-note
fix(vm-disk): the WorkloadsReady condition of a VM disk now includes the importer error reported by CDI, such as a 404 or an unresolvable host, so a disk whose source keeps failing shows why it is stuck instead of only ImportInProgress.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Workload status messages now include available DataVolume import
failure details, such as HTTP errors or DNS resolution problems, making
the cause of an import issue clearer.
* **Documentation**
* Updated the workload readiness documentation to explain when
DataVolume phase and import error details appear in status messages.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 12:29:50 +03:00
Andrey Kolkov 6a75883013 fix(backups): keep the CNPG purge for unmarked Clusters this restore did not render
A recovery Cluster without the restored-server-name annotation fell back
to the creation-time comparison. On the current chart the annotation is
absent when bootstrap.newServerName is empty, i.e. a manually configured
recovery: re-rendered during the WAL-archive wait it is created after
StartedAt, the purge was skipped and the restore could report success on
the manual recovery source. Use the creation-time fallback only when the
application's bootstrap.newServerName already carries this restore's
server name, which keeps it for Clusters rendered by a chart that
predates the annotation.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:09 +03:00
Andrey Kolkov 35ca217839 fix(backups): name the restore that goes first in ConcurrentRestore
restoreAhead returned the alphabetically smallest of the jobs ahead, so
with more than one ahead the ConcurrentRestore message could name a job
that is not the one holding the application. Return the first job in
restoreGoesFirst order, and pin the creation-time tie-break with cases
where creation order and name order disagree.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-10 23:49:09 +03:00