Commit Graph
126 Commits
Author SHA1 Message Date
Myasnikov Daniil 59932e5098 fix(platform): report FoundationDB patch drift in migration dry runs
The carry path skipped diagnostics in dry-run mode, so its summary could
report no releases needing attention even when the upgrade would lower a
FoundationDB patch version. Skip only the write and share the warning and
summary bookkeeping with a real run.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 19:14:17 +05:00
Andrei Kvapil 4cb505a512 fix(platform): record 7.1 FoundationDB clusters instead of carrying them
The operator no longer carries the 7.1 client, and the chart no longer
offers v7.1, so a 7.1 cluster has no line to be carried to. Migration
59 now records it, together with any release whose version already
names a line that is no longer offered; both have to be upgraded or
repointed before the chart will render them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil d5ebb62632 fix(platform): keep migration 60's record whole across a retry
A failed patch stops the migration before the stamp, and the retry
finds version already written on the releases it carried, so it never
reached the step that records a patch-level move. The move is now
worked out from the values alone, so a retried run records it too.

An OpenSearch release storing version: null rendered the old chart's
v2 default and is refused by the new one; it now goes on the record
before it turns up as a stuck release.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil 8c2e20539d fix(platform): record every release migration 60 moves
A FoundationDB release carried to its line still moves to the one tag
the chart ships for it, which can be a step below what it runs, and
that showed up only in the Job log. Kubernetes and KubernetesNodes
releases pointing the removed image overrides at a mirror were not
listed at all. Both now go into the needs-attention record next to the
OpenSearch ones.

The migrations image carries no charts, so the tag map is repeated in
the migration, and the bats suite fails when it drifts from the
chart's versions.yaml.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil 033bec74fb fix(platform): keep one bad release from stopping migration 60
spec.values on a HelmRelease takes any shape, so a release holding a
scalar there, or a cluster.version the apiserver stored as a number,
crashed jq and stopped the carry-over for the whole management cluster:
every healthy FoundationDB release then went into the upgrade without
its version. Those releases are now reported and recorded like any
other value that cannot be carried over.

The same record covers OpenSearch releases that still set
images.opensearch. Nothing can be carried over for them, so what the
operator needs is the list.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil f3058d2353 fix(platform): guard the FoundationDB version carry-over on all values
Migration 60 tested only cluster.version before adding version, so a
version set on the release between the scan and the patch would have
been overwritten. The patch now tests the whole values object it read.

resourceVersion is not used for this: helm-controller keeps writing the
release status, and a mismatch there would abort the platform upgrade
for no change to the values.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil 930051280b fix(platform): carry FoundationDB cluster.version over to version
The foundationdb chart no longer reads cluster.version, so a cluster
that set it to 7.1 or 7.4 would move to the v7.3 default on the first
reconcile after the upgrade: a full-bounce upgrade from 7.1, or a
downgrade from 7.4.

Migration 60 sets version to the release line of each cluster.version
it finds. It keeps cluster.version, because the old chart still renders
the release between the migration and the chart upgrade and would fall
back to its own default without it. A value outside the supported lines
is recorded on the cozystack-version ConfigMap and does not block the
upgrade.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil 873bd6712d fix(platform): stop carrying images.kubectl into adopted node pools
Migration 56 copied the parent cluster's kubectl image override into
each kubernetes-nodes release it creates. Neither chart reads an images
map any more, so the copy only planted an inert key in the pool values.

Migration 56 has not shipped in a stable release, so no cluster has run
the old mapping outside a pre-release.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:14 +05:00
Aleksei Sviridkin cbbb69afc3 fix(platform): stop treating a failed ConfigMap read as a fresh cluster
run-migrations.sh decided whether cozystack-version exists by the exit
status of a silenced kubectl get, so a timeout or a dropped connection
counted as "no ConfigMap": the runner stamped the target version and
exited 0, and every pending migration was skipped for good. The hook Job
only renders when the ConfigMap already exists, so in practice an error
was the only way to reach that branch.

--ignore-not-found suppresses NotFound alone; any other error keeps its
non-zero status and now fails the pod, so the Job retries the read.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 12:06:18 +03:00
Aleksei Sviridkin bcda5c893a feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 17:06:03 +03:00
Aleksei Sviridkin 7bf6906a80 fix(platform): keep the legacy etcd-operator stopped after etcd adoption (#4738)
## What this PR does

Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
`etcd.aenix.io` cluster with `etcd-migrate`, and then scales the
operator back to 1. At that point the Deployment still runs the legacy
operator, and after a successful adoption it has nothing left to manage.
Bringing it back undoes part of the adoption.

Observed on a production cluster upgraded from 1.4.3 to 1.6.4. The API
audit log and the Helm history of a tenant's `etcd` release show this
sequence, in UTC on the same day.

- 13:21:12 migration 50 scales
`cozy-etcd-operator/etcd-operator-controller-manager` to 0.
- 13:22:59 `etcd-migrate` updates `etcd-headless`, pointing its owner
references at the adopted EtcdMembers, and orphan-deletes the legacy
EtcdCluster and StatefulSet.
- 13:23:04 migration 50 scales the operator back to 1. It is still the
legacy operator, the v1alpha2 one replaced it only at 13:40.
- 13:23:42 Flux upgrades the tenant's `etcd` release with the old chart
(Helm revision 87), because the new chart artifact appeared only at
13:24:40. The old chart re-creates the legacy EtcdCluster at 13:23:43.
- 13:23:43 to 13:23:45 the legacy operator reconciles it, updates
`etcd-headless` twice and re-creates the legacy StatefulSet `etcd`.
- 13:36:49 Flux upgrades the release to the new chart (revision 88),
which prunes the legacy EtcdCluster.
- 13:36:50 the garbage collector deletes `etcd-headless` and the
re-created StatefulSet.

The same happened in all five tenants that run an etcd. The pods
survived because EtcdMembers own them, but they keep `spec.subdomain:
etcd-headless`, so their `etcd-<i>.etcd-headless.<ns>.svc` names stopped
resolving. From then on every member logged `no such host` for its
peers, the v1alpha2 EtcdCluster had no status conditions, and the
`etcd-defrag` CronJob failed every hour with `context deadline
exceeded`. The transitional `etcd-headless` Service the etcd chart ships
for this window (#3270) does not cover it. Helm patched the existing
Service 202 ms before the garbage collector removed it, and did not
create it again because it was already part of the release.

With this change the operator stays at 0 after a successful adoption.
The etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade, and no later migration waits for the
operator. A failed or interrupted adoption still restores it to 1
through the existing EXIT trap, so a failed run does not leave the
tenant etcds without an operator. If that HelmRelease cannot reconcile,
the adopted clusters wait for it with or without this change, because
the legacy operator does not reconcile v1alpha2 EtcdClusters.

Clusters already upgraded with this bug can be repaired without
restarting anything. Creating the Service from the tenant's own `etcd`
release is enough.

```
helm -n <ns> get manifest etcd | awk 'BEGIN{RS="\n---\n"} /\nkind: Service\n/ && /\n  name: etcd-headless\n/' | kubectl -n <ns> create -f -
```

On the cluster above the v1alpha2 EtcdCluster reported `Available` with
reason `QuorumHealthy` within 11 seconds of the Service being created,
in every tenant, and the members stopped logging `no such host`.

### Validation

- `hack/cozytest.sh hack/migration-50-etcd-adopt.bats` passes all 29
tests. A new test asserts that a successful adoption scales the operator
to 0 and never back to 1. Against the current script it fails with
`FAIL: the legacy operator was scaled back up after a successful
adoption`.
- The existing test for a failed adoption still passes, so the EXIT trap
still restores the operator to 1.
- The new test checks the absence with `if grep ...; then echo "FAIL:
..."; false; fi`, the form the other suites use. A bare `! grep`
assertion cannot fail a test under `hack/cozytest.sh`, so it could not
have caught this. The other negated assertions in this file had the same
weakness, and #3790 has rewritten them on main. With all of them made
able to fail in a scratch copy, the suite still passes against this
change.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It changes a migration script and its bats test. No package,
values, schema, CRD, make target or `hack/` layout changes, so no entry
matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): keep the legacy etcd-operator stopped after migration 50 adopts the legacy etcd clusters. Scaling it back up let it reconcile the legacy EtcdCluster that Flux re-created from a not yet upgraded etcd chart, re-create the legacy StatefulSet and take the legacy etcd-headless Service away from the adopted members, which the garbage collector then deleted. The adopted members lost their DNS names, the EtcdCluster never reported a status and etcd defrag failed every hour. Clusters already upgraded can recreate the Service from the etcd release manifest.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Migration Updates**
* After successful etcd adoption, the legacy operator remains scaled to
zero replicas for replacement by the etcd operator; it is no longer
scaled back up before the migration is stamped.
* If adoption fails or exits before completion, the legacy operator is
restored to one replica.
* The dry-run message now describes the zero-replica outcome of
successful adoption.
  * Successful adoption continues to record the migration version stamp.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-05 17:53:30 +03:00
Yan Bondarenko 6248d458ad fix(platform): keep the legacy etcd-operator stopped after etcd adoption
Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
etcd.aenix.io cluster with etcd-migrate and then scales the operator
back to 1. At that point it is still the legacy operator, and it has
nothing left to manage. Until a tenant's etcd HelmRelease reconciles the
new chart, Flux can upgrade it with the old one, which re-creates the
legacy EtcdCluster that etcd-migrate just orphan-deleted. The legacy
operator then reconciles it, re-creates the legacy StatefulSet and
rewrites the legacy etcd-headless Service, dropping the owner references
etcd-migrate pointed at the adopted EtcdMembers. When the new chart
prunes the legacy EtcdCluster, the garbage collector deletes the Service
with it, and the adopted members' etcd-<i>.etcd-headless names stop
resolving.

After a successful adoption the operator now stays at 0 until the
etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade. A failed or interrupted adoption
still restores it through the EXIT trap.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-03 18:25:49 +03:00
Aleksei Sviridkin 3a07885b70 fix(platform): resume a retried migration Job from the stamped version
The migration hook Job gets CURRENT_VERSION once, at render time, and a
failed pod is retried with that same environment. Each migration that
finished has stamped the cozystack-version ConfigMap by then, yet the
retry started from the rendered version and ran those migrations again.
Migrations are written to run once, so a re-run is not safe in general.

The runner now starts from the version stamped in the ConfigMap. A
failed read aborts the pod, since falling back would bring the re-run
back and the Job retries anyway. The rendered value is kept only when
the ConfigMap holds no integer version, which is also the case where
the chart could not derive a real one from it. An all-digit stamp past
the shell's integer range aborts too: its comparison fails inside an
if, out of reach of set -e, and the Job would go green having run
nothing, while the chart renders that stamp as 0.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 21:54:46 +03:00
Aleksei Sviridkin fd8ccb2fcb feat(platform): keep VMI-only hotplugged volumes across declarative hotplug
Dropping the HotplugVolumes gate lets virt-controller rebuild each
running VMI's hotplug volumes from VM.spec, and it removes every
hotpluggable VMI volume the VM spec does not list. kubevirt-csi
attached tenant volumes at the VMI level until mid-2025 and live
migration carries such a VMI across KubeVirt updates, so a tenant node
that was never recreated can still hold a mounted disk only its VMI
knows about; the first reconcile after the upgrade would unplug it.

Migration 59 writes each such volume and its disk into the VM template
as they are on the VMI, before the upgrade re-points cozystack-packages
and so before the KubeVirt CR and the kubevirt-csi flag change. Hotplug
volume changes are a valid live update, so no VM needs a restart.

An ephemeral hotplug a user made at the VMI level, such as virtctl
addvolume without --persist, has the same shape, because virt-api gives
every VMI-level addvolume that shape. It is copied too and becomes
permanent: leaving it out would not keep it ephemeral, since the first
reconcile would unplug it. Only Running VMIs are considered, matching
virt-controller, which reconciles no other; a stopped VM's next start
builds its VMI from the VM spec, so its VMI-only volumes are not
resurrected.

Only the variants with migrations enabled run it. The default variant
renders no migration hook, so a cluster there that installed KubeVirt
by hand has to check for such volumes itself before upgrading.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-28 23:37:48 +03:00
renovate[bot] a7dfbe6f8c chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-24 16:55:04 +03:00
Timofei Larkin a3911811c7 feat(api): take Application name limits from the application's schema
The API server capped Kubernetes and Kafka names by knowing what those
two charts render: worker pool releases under a 17-character prefix,
controller pods with a 13-character suffix. That is the charts'
knowledge sitting in the platform, and it does not generalise. The next
application with a sub-resource budget needs another branch on Kind,
and an application served from outside this repository cannot get one
at all.

An application now declares the limits on its own name in its schema,
under x-cozystack-name, and the server applies whatever it finds there
for whatever kind the ApplicationDefinition describes. A declaration
only tightens: prefix + name must still fit the 53-character Helm
release name limit. A rejected name is answered with the chart's own
explanation of the limit.

The schema is untrusted input, and definitions from other repositories
are written by hand. A declaration that cannot be enforced as written,
whether an unknown keyword, a wrong type, a limit no name satisfies or
a pattern Go cannot compile, refuses creates of that kind with a
configuration error. Dropping it would admit names the chart cannot
render; applying it would reject every name and blame the user.

An upgrade brings cozystack-api up before the definitions it serves are
re-rendered, so for a while the new server can read a Kubernetes or
Kafka definition that predates its declaration. The two old caps stay
as a floor for exactly that case, a definition that declares nothing,
until no supported upgrade starts from one.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-09-23 10:59:51 +03:00
Timofei Larkin c398e0e116 Merge branch 'main' into chore/drop-http-cache
main took migration 57 and moved migrations.targetVersion to 58 while this
branch was open, so that line is the one conflict: 57 at the merge base, 58
on main, 59 here. Resolved as 59, which is what this branch's migration 58
requires, since run-migrations.sh loops up to targetVersion - 1.

Merged rather than rebased because the branch carries commits that are not
mine to rewrite.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-09-18 13:20:03 +03:00
Timofei Larkin 2cbc02bd10 fix(platform): freeze HTTPCache on the artifact the cluster runs
Pinning the frozen source to a digest written into the migration picks one
release for every cluster, and v1.6.0 is not even the last one that ships
http-cache: v1.6.1 through v1.6.3 and v1.7.0-alpha.1 all do, and 1.6 patch
releases may keep doing so. A cluster on any of those would be rolled back
to an older chart by the upgrade that was meant to leave its instances
alone, and an air-gapped mirror resolves the constant only if someone
mirrored that exact manifest.

The hook runs before the upgrade re-points cozystack-packages, so the live
object still names the release the instances render from, in whatever
registry the operator serves it from and with whatever pull secret and
verification they configured. Reading the digest off it makes the freeze a
no-op by construction and needs no knowledge of which tag was last.
spec.ref.digest is used when the installer pinned one; a tag or semver ref
goes through status.artifact.revision, whose digest half is the manifest
digest. status.artifact.digest is the checksum of the downloaded tarball
and cannot stand in. A source with neither has nothing to freeze on and the
migration stops there. The copy also carries the no-delete label the
platform chart puts on the original, because it is the only chart source
those instances have left.

The tenant grant travels with the freeze as well. cozy:tenant:admin:base
enumerates the apps.cozystack.io resources a tenant owner may write, and an
application that has left the catalog has no business in that list: the
next sweep over it takes the entry as dead config, and the failure is
silent and remote, a Forbidden for the one person who still needs to remove
what they have. The chart drops httpcaches, and a cluster that freezes gets
a ClusterRole under the tenant admin aggregation label instead, applied
before the upgrade drops the rule so the access never lapses and carrying
the verb set the chart had so nothing changes for the owner. A cluster with
no instance is granted nothing.

The Package delete on the no-instance path has to block. The operator
reconciles off an informer cache, so a reconcile in flight against the
pre-delete entry re-creates the release it owns, and with it the
ApplicationDefinition and the Helm storage the following lines remove.
Returning immediately made the ordering a property of the command log
rather than of the cluster.

The preset check's SIGPIPE note named http-cache-rd as its example of a
large enum, and that chart goes away here; kubernetes-rd carries the same
enums.

Numbered 58 rather than 57: main already carries a migration 57, the
SeaweedFS volume-size-limit one, and its targetVersion is already 58.

The whole shape of this retirement was worked out here rather than
anywhere reusable, so docs/deprecating-applications.md writes it down as a
procedure: how to pick between migrating onto a successor and freezing,
what the prune takes with it, and what the migration owes a cluster on
each branch. The next application to go should not have to rediscover it.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-09-18 13:12:56 +03:00
Aleksei Sviridkin a2db2aeb8c fix(seaweedfs): raise the number of buckets a volume server can serve
A volume server started with -max 0 takes a slot count derived from disk
rather than a fixed one, replication 001 spends a slot on each of the
two servers per logical volume, and a collection claims COPY_2 volumes
whenever it grows. So the bucket ceiling is set by the product of the
size limit and COPY_2, and moving either number alone moves it. The
mechanism is upstream in MaybeAdjustVolumeMax (weed/storage/store.go)
and findVolumeCount (weed/topology/volume_growth.go).

Do not divide these out of the nominal PVC size, because the count
follows the space the filesystem makes available and not the volume's
advertised size: the shipped 10Gi volume PVC was seen to give nine slots
per server at the upstream 1000 and 19 at 500, which puts 400 at 23 to
24 by the same division, or eight collections against two. The span is
the observation's own, since the free space behind 19 at 500 fits more
than one filesystem geometry and those divide differently at 400; eight
either way, because a collection that cannot get a whole batch of COPY_2
keeps the volumes it did get and writes into them. The division itself
is measured: on a 1 GB image the master reports max as the filesystem's
available space over the limit in every run, 25 slots and nine
collections at the 40 and 3 that scale to this pair. Two is what the
platform's own cozy-backups bucket and a single tenant bucket already
use. Past that every S3 PUT to a new bucket times out while the disk
still reads as nearly empty, and no COSI object reports anything wrong,
because the claim and the access are both granted.

400 and 3 make the budget 1200 MB per collection. Splitting the
reduction across both numbers rather than taking it out of the size
limit alone keeps the cost off the curve that grows: open descriptors
track stored bytes divided by the limit, so a 10 TiB server keeps
roughly 52k of them at this limit against 21k upstream, where moving the
size limit alone would have put it at 105k. A .dat and an .idx stay open
per volume; the .vif beside them is written and closed. The ceiling
reached is the same either way.

Both figures describe the shipped replicationFactor of 2. The master
picks its growth count from the replication the filer asks for, so at 1
and 3 copies it reads COPY_1 and COPY_3 instead and this COPY_2 does not
apply; lowering the size limit still raises the ceiling there.

The size limit was out of an operator's reach: it lived in the
data-plane chart, which is pulled by ExternalArtifact from a component
that declares no install section, so a Package-level values override for
it is dropped before any merge. The app chart now declares
master.volumeSizeLimitMB and sends it on every release, and 400 is set
there. The data-plane chart keeps upstream's 1000 as its fallback on
purpose. A platform upgrade can re-render that release from its artifact
before the parent has sent anything new, and a lower fallback would then
reach the master first and drop the limit ahead of an operator's own
setting. COPY_2 is sent by nothing and lives in the data-plane chart, so
it does land in that window, where a grow costs 3000 MB against the 6000
MB it cost before.

The app chart also writes maxVolumes into dataDirs on every release, and
because dataDirs is a list Helm replaces the data-plane chart's copy
wholesale, so the zero that makes the slot count auto-derived at all
comes from there. Losing it drops the server to the subchart's fixed
seven without failing any render.

The field is bounded at both ends, because either end takes an instance
down. Zero is not a smaller limit but an off switch: the volume server
leaves its slot calculation untouched at zero, keeps reporting no
capacity, and nothing is ever grown. Past 30000 the master does not
start at all. The bound sits in the values schema rather than in a
template guard, so Helm refuses the value before any template runs. The
apps API does not: it reads that schema only for defaults, so it accepts
a value out of range, and the release then fails to render. An explicit
null gets past both bounds, because Helm drops the key before it
validates, and would render an empty -volumeSizeLimitMB= that the master
exits on at start, measured on the shipped image, so the template
requires the value.

No bound against the disk goes with it, though the field invites one. A
limit at or above the volume's own size does not break an instance:
MaybeAdjustVolumeMax floors the slot count at one, measured on 4.31 with
a 255 MiB disk against a 1000 MB limit, which reported Max 1 and served
a write it then read back. Such a server is capped at a single slot
rather than dead, so refusing it at render would be a capacity policy
that stops a working release on every reconcile, not only on the
upgrade. The field's description carries the relation instead, where the
value is chosen.

The growth count is deliberately not exposed beside the limit. The
master picks which of its four growth counts to apply from the
replication the filer requests, and the filer takes that from
replicationFactor, so a field wired to COPY_2 would do nothing at 1 or 3
copies while still appearing to be a setting. A knob that is silently
inert for two of the three supported values is worse than no knob.

Two further prices. A collection absorbs the budget before it needs
another grow, so growth requests arrive five times as often as at the
upstream pair. A collection also starts with half as many writable
volumes, and a volume server takes the writes into each volume one at a
time (syncWrite holds that volume's dataFileAccessLock), so a heavily
parallel writer spreads over fewer volumes until the collection grows
again; that holds for instances kept at 1000 too. And lowering the limit
on an instance that already holds data carries the other one: its
volumes between the new limit and the old go oversized on the next
heartbeat and leave the master's writable set, and each affected
collection needs one successful grow before its next write lands.
Nothing is lost and the volumes stay readable; the room left inside them
for further writes is given up and does not return as capacity.

Those grows compete for one pool, and the pool is smaller than it looks.
An oversized volume is charged no headroom, while one below the limit is
charged all of its unfilled room; a grow then claims a whole COPY_2
batch and takes as much of it as is there, and the volumes it creates
are empty and charge their headroom straight back. Measured on a local
SeaweedFS 4.31 setup: two oversized collections with two free slots
between them, the first to write took both, and the second stayed
refused with the disk still reporting a quarter of itself free.
Collections that need no grow are not free either, since their volumes
below the new limit were created six at a time and each can hold back a
whole limit. Also measured there, with the limit and batch scaled down:
a server with free space for one batch per collection refused the grow
its one affected collection needed, because a second, nearly empty
collection held the room, and the same free space without it let the
grow through. The minFreeSpacePercent reserve only comes into it on a
server whose writable volumes have all passed the limit, where nothing
is charged headroom: from 8000 MiB of filesystem upward, a 10Gi PVC
measuring 9979, five percent is at least 400 MB, and such a server then
refuses a grow only once its disk is down to the reserve. Everywhere
else the headroom charge decides, and a refusal can arrive with the disk
far from full whatever the reserve. Deleting a collection gives the room
back a slot per volume, however little it stored, since the headroom
charged to those volumes leaves with them.

So, on a variant that runs platform migrations, the upgrade does not
change the limit on an existing instance unless its operator already set
one, which the previous chart accepted and ignored. Platform migration
57 writes master.volumeSizeLimitMB: 1000 into every existing SeaweedFS
instance that has not chosen a limit, and 400 applies only to instances
created from here on. It leaves alone only an instance whose master is
not an object, which no v1 chart renders, and names that one in its log.
The default variant runs no platform migrations, so a cluster there that
installed SeaweedFS by hand gets no pin, and the document below gives
the same patch to apply before upgrading. A chosen limit is a whole
number the new chart accepts; anything else counts as not chosen and is
replaced with 1000, which is what the previous chart ran in its place,
since it ignored the key and the new chart would refuse to render it. A
valid value already set is kept, and since the previous chart ignored
it, the upgrade is when it starts to apply; the document below lists the
ones that change the limit ahead of it. Raised values are listed as
well: existing buckets keep writing, but every volume below the new
limit is charged up to it, and on a local SeaweedFS 4.31 setup seven
nearly empty volumes left no free slot once the limit went up tenfold,
so a new bucket was refused. The value goes into the application's
HelmRelease, whose spec.values is the application spec, for an instance
created through the API and for one the tenant module renders alike. The
tenant module never renders spec.values on that release, and a Helm
upgrade leaves a key alone that neither manifest carries: measured with
Helm 4.3.0 on client-side, server-side and forced server-side apply, and
read in createPatch at the v4.1.1 that helm-controller v1.5.0 runs.
Where the migration runs, it is a pre-upgrade hook of the platform
release, and the cozystack-packages source every application chart comes
from is a manifest of that same release, applied only after its hooks;
so no instance meets the new chart unstamped, and a failed migration
stops the platform upgrade before that source moves. That holds while
the platform source is pinned by digest, as it is out of the box; one
that follows a tag, a semver range or a branch lets source-controller
move cozystack-packages on its own schedule. Until cozystack-packages
moves, each patched release keeps its old chart, which ignores the key,
since no v1 release reads it or closes the master object in its schema.
An instance kept at 1000 still gains, because at a replication factor of
2 a grow now claims three volumes rather than six, so it has room for
roughly twice as many new buckets.

Lowering an existing instance is left to its operator, and
docs/operations/seaweedfs-lowering-the-volume-size-limit.md is the check
for it. It lists each server's volume files and adds one limit for every
volume below the limit and a batch for every collection with no volume
at or below 90 percent of it, since the master grows such a collection
on its own schedule without a write; measured the same way, a collection
with every volume in that band and no writes got two new volumes per
server within one pass. Where each disk type is served by as many volume
servers as the replication factor, as the shipped two at a factor of 2
are, each collection's volumes sit on every server of that type, and
that bounds the claim from above without assuming how the collections
grew. The master places a new volume on any server of the requested disk
type with room rather than where its collection already lives, whichever
group or pool that server belongs to, and it reads an empty disk type as
hdd; under MultiZone it also grows each zone on its own. So the document
keeps zoned instances, and any disk type served by more servers than
copies, at 1000.

BREAKING CHANGE: on the default variant, which runs no platform
migrations, set master.volumeSizeLimitMB to 1000 before upgrading on
every SeaweedFS instance that has no valid value and whose master is an
object, with the step in
docs/operations/seaweedfs-lowering-the-volume-size-limit.md. On the
variants that run platform migrations, migration 57 does this. If the
step is skipped, every such instance drops to 400 on the upgrade: each
of its volumes at or above 400 MB leaves the master's writable set, and
a bucket whose volumes have all crossed that line takes no writes until
it can grow.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-17 00:54:48 +03:00
Andrei Kvapil 4953b512a5 chore(platform): freeze HTTPCache on the last artifact that ships it
The earlier shape of this migration orphaned every instance into plain
objects, which left its owner without a way to manage or remove it. A
cluster that still runs HTTPCache instead keeps the application, served
from a copy of the packages OCIRepository pinned to the digest of the
last release that contains it. The copy carries no owner references and
no Helm metadata, and the PackageSource is repointed at it, annotated
keep and disowned, so neither Helm nor the operator prunes it. Helm
reads that annotation off the live object during the upgrade prune,
which is what makes disowning in a pre-upgrade hook work at all.

Names are deliberately unchanged: the ApplicationDefinition inside that
artifact hardcodes its chartRef, so renaming the PackageSource would
leave every release pointing at an artifact nobody produces.

A cluster with no instance keeps nothing. The naas bundle installs the
rd component regardless of use, so freezing unconditionally would strand
a pinned source on clusters that never touched the application.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-16 11:00:54 +02:00
Andrei Kvapil c0f9f08de7 chore(platform): detach HTTPCache instances before the package is pruned
Pruning the http-cache PackageSource takes its chart artifacts with it,
so without a migration every http-cache HelmRelease loses its chart and
stays Ready=False for good, with HelmReleaseNotReady firing on every
cluster that runs the naas bundle. Deleting the leftover Package by hand
is worse: it takes the ApplicationDefinition with it while the pods of a
surviving instance are still running.

Migration 57 orphans each tenant release instead, the way migration 55
does, so the workloads stay in the tenant namespace as plain objects,
and then removes the platform side. The Package goes before
hr/http-cache-rd because the operator re-creates a release it owns for
as long as that Package exists.

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-09-16 00:34:51 +02:00
IvanHunters 32ec1b5a00 fix(migrations): correct unadopted-record count semantics and stale comments
- The unadopted-record WARNING said "N pool(s)" but counted SKIPPED_FILE lines,
  and the foreign-owner branch writes one line per object (so a single pool with
  MD+MHC+WM reported "3 pools"). Reword to "N record(s) ... one line per object
  or pool ... not a distinct-pool count" so the number does not mislead.
- The inline-chart restamp comment implied the cozystack-apps source is enforced;
  it is not (matching by chart name only is a deliberate trade-off: gating on the
  historical sourceRef risks under-matching a real legacy parent, which loses its
  workers, whereas the loose match's worst case is a harmless mislabel). Spell out
  the trade-off and that the render guard is the data-loss backstop.
- The overflow comment cited "up to 42 chars per rest.go"; the enforced cap is 32
  (maxKubernetesClusterName = 53 - 17 - 4). Correct it and note legacy/hand-created
  names can still be longer, which is why the overflow branch exists.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 14:40:20 +03:00
IvanHunters c0fcdd767c fix(migrations): discover pre-chartRef parents via inline spec.chart in the restamp
The restamp discovered unlabelled parent kubernetes releases by spec.chartRef
only, but that misses the population it exists for. The apps.cozystack.io/
application.kind label was introduced 2025-12-19 (fdca49838); spec.chartRef only
2026-01-14 (43da779ee). So any parent new enough to carry chartRef already
carries the label and is handled by the label-selected sweep, while every
genuinely unlabelled parent predates chartRef and instead carries an inline
spec.chart (chart name "kubernetes" from the cozystack-apps HelmRepository).
The chartRef-only selector therefore matched none of the real targets. Match both
shapes. This is a seamless-adoption improvement, not a data-loss fix: the chart's
own render-time guard already fail-closes (blocks the upgrade rather than pruning)
for any parent the restamp misses. Exact chart-name match plus the kubernetes-
name-prefix guard keep it from touching other apps or kubernetes-nodes children;
covered by an inline-shape restamp test and a non-kubernetes decoy test.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 13:50:14 +03:00
IvanHunters c6b6755eec fix(migrations): record foreign-owner skips, harden record_skip, scope restamp to kubernetes- names
Three hardening fixes on migration 56's new code:
- The foreign-owner branch in adopt_one pins the object prune-proof but does not
  adopt it (the child release can then stay Failed on the ownership conflict),
  yet it was the one warn+pin+skip path that did not record itself in the durable
  unadopted ConfigMap -- so an operator relying on that record to find stuck
  pools would miss it. Record it like every sibling branch.
- record_skip ran a bare append under set -e inside skip branches that are
  designed never to abort; a failed write (e.g. /tmp full) could deadlock the
  fleet upgrade for one tenant. Make it fail-open, matching the branches' intent.
- The restamp selector matched any release by chartRef; scope it to names
  starting with kubernetes- so it cannot mislabel (and then have the sweep skip)
  a non-standard release, keeping restamp and the sweep's contract in sync.
Also fix the previously-vacuous negative restamp test (its fixture lacked the
chartRef, so it never reached the label guard it claimed to check).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 13:28:58 +03:00
IvanHunters 3d1a83f419 fix(migrations): make the unadopted-pools record publish best-effort
The durable-record ConfigMap write sat under set -euo pipefail right before the
version stamp, so a permanent failure (e.g. the record exceeding etcd's object
size limit at a very large skip count) would abort the migration and deadlock
the fleet upgrade with no path out -- even though every skipped pool was already
pinned prune-proof earlier in the sweep, so the workers are safe regardless of
whether the record lands. The record is a convenience artifact, not a safety
mechanism, so swallow its failure with a warning instead of failing closed.
Covered by a bats case that fails the publish and asserts the run still stamps.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 12:38:28 +03:00
IvanHunters 952901776a fix(migrations): adopt unlabelled parent kubernetes releases and record unadopted pools in migration 56
The worker-adoption sweep selected parents by the apps.cozystack.io/
application.kind label, which postdates v1.2. A parent kubernetes HelmRelease
older than that, or one restored by hand, lacks the label: the sweep skipped
it, stamped, and the parent's control-plane-only re-render then pruned its
un-pinned workers with no pin and no warning. Discover such releases by their
chartRef instead and restamp the three application labels before the sweep
(additive, cannot corrupt a release, and repairs it for cozystack-api too).

Every warn+pin+skip branch leaves a pool pinned prune-proof but unadopted,
recorded only in the Job log, which the next upgrade reaps; the migration does
not re-run, so an operator cannot find those pools afterward. Accumulate the
skips and publish them as configmap/cozystack-migration-56-unadopted before
the stamp.

Extend the fake kubectl and add bats coverage for the restamp (positive and
negative) and the unadopted-ConfigMap paths.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 12:28:46 +03:00
IvanHunters 5b08d48632 docs(migrations): pin DEFAULT_MD0 provenance to the pre-removal helper commit
The kubernetes.nodeGroups helper that DEFAULT_MD0 reproduces was deleted in this
line of work, so nothing in-tree diffs against it and a future edit to the
constant (instanceType/diskSize/...) would silently rename the content-hashed
KubevirtMachineTemplate and roll every implicit-md0 pool. Reference the
pre-removal definition (git show 92b9785da^:.../kubernetes/templates/_helpers.tpl)
so the canonical default stays diffable.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-24 11:26:44 +03:00
IvanHunters f035da8a0f Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve conflicts introduced by main:
- Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in
  favour of the GPU type main added; main removed the fluxcd addon wholesale.
- Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's
  redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57.
- Accept the branch's removal of the in-chart talos-reconcile Job; it moved to
  the kubernetes-nodes chart, main only touched its comment.
- Regenerate kubernetes values.schema.json / README and the kubernetes-rd
  ApplicationDefinition schema.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-20 14:04:32 +03:00
Andrei Kvapil 851a86dcf1 feat(platform): orphan tenant FluxCD HelmReleases on upgrade
Add migration 54 for the removed addons.fluxcd feature. For every tenant
that had it enabled, the FluxCD HelmReleases in the management cluster
are suspended, stripped of their Flux finalizer and deleted, so the
delete does not trigger a Helm uninstall inside the tenant cluster. The
Flux release already running in the tenant keeps running untouched.

Runs as a pre-upgrade hook, before the addon-less Kubernetes chart
reaches the tenant HelmReleases. Mirrors the orphan idiom of migration
29. Bumps migrations.targetVersion to 55.

Fail closed. Dropping the finalizer is the only thing that keeps the
delete inert, and migrations never re-run, so the script runs under
`set -euo pipefail` with no `|| true` on the critical path and gates the
delete on the finalizer actually being gone. The verify read uses
`--ignore-not-found` rather than `2>/dev/null || true`, so a failed read
aborts instead of reading as an empty finalizer list that passes the
gate on an object whose state was never checked. A migration that stops
is recoverable; one that half-orphans a fleet is not.

Cover it with a docker + fake-kubectl unit test running the real script
by path inside the migrations image's own alpine base (busybox ash, jq
baked on), so it exercises the shipped interpreter and load-bearing
pipefail. The suite pins ordering and selection (only the addon
HelmReleases are touched, each suspend -> finalizer-drop -> delete in
order) and the fail-closed paths (a surviving finalizer, a failing
verify read, and a failing fleet scan each abort before deleting and
before stamping).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 13:37:09 +03:00
IvanHunters cd72d81bb4 Merge branch 'main' into feat/kubernetes-nodes-phase2b
Reconcile the phase-2b split with main's configurable worker-health
feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/
podCpuRequest), which landed on both charts after this branch forked:

- kubernetes (parent): pool rendering stays removed; main's parent
  worker-pool additions do not apply post-split.
- kubernetes-nodes (pool): absorb the feature; keep the directional
  parent-version skew guard and the immutable cluster/storageClass rules.
  KubevirtMachineTemplate content hashes are unchanged (render snapshot).
- migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the
  cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime
  into adopted pools, so adoption neither rolls nor resets a tuned pool.
  Covered by a new adopt-values bats case.
- drop the parent chart's obsolete MHC-timeout suites; the
  kubernetes-nodes twins cover the feature.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-19 13:14:11 +03:00
Andrei Kvapil c7a3320bb4 fix(migrations): pin the engine and fail-close the CRD probe in migration 54
Two defects in migration 54's adoption pass:

(1) The label landed before the image pins, so the operator adopted a
labelled CR with empty image fields and rolled Redis to the fork's 7.2.4
default — rewriting dump.rdb into a format the later 6.2.6 chart pin can no
longer load. Pin spec.redis.image/spec.sentinel.image to redis:6.2.6-alpine
in the same pass, only where empty.

(2) The CRD probe used a bare `get crd >/dev/null 2>&1` that could not tell
NotFound from an API blip and stamped anyway; switch to migration 50's
`--ignore-not-found -o name` so a real error aborts under set -e.

Also self-contained the renumber note and qualified the resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-19 10:15:59 +02:00
Andrei Kvapil 0a3a3bfa77 fix(migrations): adopt existing RedisFailovers onto the operator group
The freshworks-oss operator watches RedisFailovers through a server-side
operator-group label selector, so a CR without the label is silently dropped
from its informer. The chart templates now stamp the label, but an existing CR
only gets it when its own HelmRelease next upgrades — after the operator rolls
— leaving a window with no healing, and a suspended/stuck HelmRelease never
gets it at all. Label every RedisFailover in a pre-upgrade migration, ahead of
the new operator.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-19 10:15:58 +02:00
IvanHunters f9065a9455 fix(migrations): correct the non-scalar-field skip remedy
The non-scalar scalar-field skip branch told the operator to restore the field
and 're-run adoption', but migration 54 stamps the version unconditionally and
does not re-run, so that remedy is a dead end. Align it with the sibling skip
branches: restore the scalar, then recreate the pool, and state plainly that
the migration has already stamped and will not re-run.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-14 17:21:14 +03:00
IvanHunters 8815468c9b fix(migrations): route non-scalar nodeGroup scalar-fields through warn+pin+skip
Migration 54 copied per-group scalar fields (storageClass, instanceType,
diskSize, minReplicas, maxReplicas, logSerialConsole, maxUnhealthy,
nodeStartupTimeout) verbatim into the child HelmRelease values. The nodeGroups
map value was never fully schema-constrained, so a kubectl/dashboard edit could
store one of these as an object or array; copied through, it fails the
KubernetesNodes schema on the child install and leaves that child release stuck
Failed.

Validate the scalar fields before building the child values and route a
non-scalar value through the same warn + pin + skip contract as the invalid
name, overflow and non-object branches: pin the pool's objects prune-proof and
skip adoption, so the pool keeps running unmanaged until the operator restores
the stored value. Container-typed fields (roles, resources, gpus, kubelet) are
left untouched.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-13 18:44:07 +03:00
IvanHunters 9ec5f8ceca fix(migration): carry logSerialConsole through worker-pool adoption
Migration 54's adopt pick-list copied every worker NodeGroup field that feeds
the content-hashed KubevirtMachineTemplate except logSerialConsole. That field
renders into the same hashed template (kubernetes-nodes nodegroup.yaml), so an
adopted pool that set logSerialConsole would render a KMT without it, change the
content-hash, rename the KMT, and roll every worker VM of the pool on upgrade
(and silently drop the setting). Add logSerialConsole to the pick-list (guarded
by has(), so pools without it are unaffected) and assert it carries through.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-11 10:56:07 +03:00
IvanHunters e969d71379 chore(platform): use bash shebang for migration 54
Migration 54 uses set -o pipefail, a bashism. The migrations image ships
bash (alpine:3.24 + apk add bash) and sibling migrations 45/47/50/51 already
use #!/bin/bash for the same reason. Align 54 so the interpreter matches the
options it sets and cannot silently no-op pipefail if /bin/sh were ever a
non-bash shell.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 23:02:36 +03:00
IvanHunters cb56f140ad fix(platform): pin keep on a foreign-owned worker object before skipping it
The foreign-owner skip branch warned and returned without applying
helm.sh/resource-policy=keep, on the premise that a foreign release-name
means the parent will not prune the object. That premise is unsound: Helm
computes the upgrade prune set from the release's own stored old manifest
minus the new manifest and gates deletion only on resource-policy=keep, not
on the live meta.helm.sh/release-name. The worker object is in the parent's
old pre-split manifest, so the control-plane-only re-render would prune it
and cascade-destroy the tenant's worker VMs.

Pin keep before returning, exactly as every sibling skip branch does, while
still refusing to rewrite the foreign release-name (which could corrupt that
release). Also correct the printed remedy: migration 54 stamps 55
unconditionally and never re-runs, so recovery is manual, not a re-run.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 22:19:35 +03:00
IvanHunters 4f323bfce5 fix(platform): pin live worker pools when migration-54 sees non-object nodeGroups
The top-level non-object nodeGroups branch warned and skipped the cluster
without pinning any objects, unlike every sibling skip branch (RFC-1123
invalid key, 53-char overflow, per-group non-object) which pins first. If a
cluster's pools were rendered under an earlier good revision and its
nodeGroups was later mangled out-of-band into an array/string/number, the
worker objects stay live but un-pinned; the parent's control-plane-only
re-render then prunes them, destroying the tenant's running worker VMs.

The declared pool keys are unreadable in this shape, so discover the live
pools from their KubevirtMachineTemplates (kubernetes-<cluster>-<group>-<hash>)
and pin each via pin_pool_objects before skipping adoption. Fail-closed
through list_kmts so a transient read never masquerades as no-pools.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 21:52:40 +03:00
IvanHunters 2d73043518 fix(platform): scope migration-54 foreign-owner refusal to one object
A worker object whose meta.helm.sh/release-name is neither the parent nor
the child release made adopt_one exit 1. Under set -e that aborts the whole
per-cluster pre-upgrade hook, so run-migrations.sh never stamps the version
and one tenant's out-of-band-edited annotation deadlocks the platform
upgrade for every tenant, with no path out.

This is the same hostile-stored-shape class the non-object, overflow and
RFC-1123-invalid branches were already reshaped to warn-and-skip rather than
deadlock the fleet. A foreign owner also means the parent release no longer
owns the object, so its control-plane-only re-render will not prune it:
skipping is prune-safe. Refuse to adopt (rewriting a foreign release-name
could corrupt that release) but scope the blast radius to the single object:
warn, skip it, document the operator remedy, and let the run complete.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-03 21:51:28 +03:00
Timofei Larkin 730d006ce8 fix(platform): skip pinning names no object can have
The pin path composed "kubernetes-<cluster>-<group>" and read it back,
but a group key that cannot form a valid object name (a slash, a space,
an underscore...) provably never had objects applied under it -- the
old chart's apply would have been rejected by the apiserver. Worse,
kubectl rejects a name containing '/' client-side, deterministically,
so pin_keep's fail-closed read classified it as a real failure and the
migration exited 1 on every retry: a permanent fleet-wide pre-upgrade
deadlock over a pool that cannot exist.

Lean on the invariant instead of the read: pin_pool_objects statically
checks the composed name against the DNS-1123 subdomain shape and skips
with a note when it cannot name an object -- nothing to pin, no kubectl
call to misread. Keys that are label-invalid but still valid object
names (e.g. dotted) may have live objects and keep today's pin
behavior; the pin-path transient-error test now uses a dotted key so it
still exercises a genuine read.

The fake kubectl reproduces real kubectl's client-side slash rejection,
and a new bats case pins the skip: a "pool/a" key must warn, pin
nothing, apply nothing, and still reach the version stamp. Red against
the previous script, which deadlocked at the first pin read.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-25 02:40:10 +03:00
Timofei Larkin bdadb3ff04 fix(platform): detect absence via --ignore-not-found
The stderr-based classification this branch introduced had two holes of
its own. Capturing stderr into the data variable (2>&1) contaminates
the SUCCESS path: kubectl prints warnings on successful reads (API
deprecation headers -- deterministic for a deprecated GVK, and CAPI
v1beta1 is one -- plus client-side throttling notices), so the warning
glued onto the owner annotation would trip adopt_one's foreign-owner
refusal on every tenant and permanently deadlock the pre-upgrade hook.
And the *"not found"* pattern over-matches: a transient
conversion-webhook outage fails a read with '... service
"capi-webhook-service" not found', which the classifier would treat as
absence -- skipping the keep pin and re-opening the exact fail-open
prune hazard this branch exists to close.

Detect absence with --ignore-not-found instead: a missing object is
SUCCESS with empty output, so a non-zero exit can only be a real
failure and aborts before the version stamp, with no stderr parsing at
all. adopt_one reads -o json and extracts the owner annotation with jq
(already a dependency), keeping absence distinct from present-but-
unannotated; pin_keep reads -o name. list_kmts still needs stderr for
the one legitimate classification -- the two deterministic missing-CRD
messages -- so it captures stderr to a temp file, never into data, and
no longer accepts NotFound-shaped text for a list.

The fake kubectl honors --ignore-not-found the way real kubectl does,
gains FAKE_WARN_STDERR (success-path stderr warnings) and
FAKE_GET_FAIL_MSG (transient error with arbitrary text). Two new bats
tests, each red against the previous commit: a deprecation warning must
not break adoption or fake a foreign owner, and a failure whose message
contains "not found" must still abort without stamping.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-25 02:07:28 +03:00
Timofei Larkin f8258c8859 fix(platform): fail migration 54 reads closed
kubectl exits 1 for NotFound and for transient server failures (timeout,
429, etcd hiccup) alike, and migration 54 treated every non-zero read as
"object absent". A transient error while reading a worker object would
skip the keep pin and the ownership rewrite, let the migration complete
and stamp version 55 (so it never re-runs), and the parent's
control-plane-only upgrade would then prune the un-pinned live
MachineDeployment -- deleting every running worker VM in the pool. The
doubly-masked KubevirtMachineTemplate listings (2>/dev/null plus
`|| true`) had the same shape: a failed list adopted or pinned zero
KMTs, permanently stranding a preserved older-revision KMT still
referenced by an in-flight MachineSet, since the child chart only
re-emits KMTs already annotated with its own release name. The
`cur=$(... || true)` owner read additionally bypassed the foreign-owner
refusal guard.

Classify reads by stderr, the migration-47 precedent: NotFound means
genuinely absent and is skipped; anything else exits 1 so
run-migrations.sh leaves CURRENT_VERSION at 54 and retries on the next
upgrade. adopt_one and pin_keep fold the existence check into the
annotation read (one request instead of two); KMT listings go through a
capture-first list_kmts helper because `for x in $(...)` discards the
substitution's exit status even under set -e, leaving `|| true` to
cover only the empty-grep-match case, mirroring the unpin hook. The
child-HR existence check stays deliberately tolerant: its fallback is
an idempotent apply that itself fails closed. adopt_one now also stamps
meta.helm.sh/release-namespace so an object whose Helm annotations were
stripped out-of-band cannot fail the child's ownership validation.

The fake kubectl now reports absent objects the way the real apiserver
does (NotFound on stderr, exit 1) and gains FAKE_GET_FAIL to inject
transient errors; four new bats tests pin the classification: NotFound
still completes and stamps, while a transient error on the adopt path,
the KMT listing, or the pin path aborts before the version stamp. Each
red against the previous script.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-25 01:45:32 +03:00
IvanHunters 928df3c2d3 fix(platform): harden migration 54 against non-object and whitespace nodeGroups
The RFC-1123 key guard added in the previous round closed hostile map
keys, but migration 54 still trusted the shape of nodeGroups and of each
group value, and word-split keys on whitespace. Because it runs as the
platform pre-upgrade hook, either gap can deadlock the upgrade for every
tenant off one hand-mangled parent HelmRelease (kubectl edit bypasses the
aggregated API schema, the same vector the key guard addresses).

Shape: a non-object nodeGroups (array/string/number) crashes jq under
set -e (keys[] on an array yields indices, then .[$idx] fails); a
non-object group value aborts the value-mapping jq on `$grp | has(...)`.
Normalize both: skip the cluster for a non-object nodeGroups map, and
route a non-object group value through the same warn + pin + skip path as
an invalid name. {} / null still fall through to the implicit md0 default.

Whitespace: an unquoted `for group in $(... keys[])` word-splits (and
globs) a key like 'my pool' into fragments that each pass the RFC-1123
guard, fabricating garbage child releases. Iterate keys newline-safely so
the intact key reaches the guard. Fail-closed exit semantics are
preserved through the while pipeline.

Covered by three new bats fixtures in migration-54-adopt-path.bats, each
red against the current script and green with this change.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-24 15:33:26 +03:00
IvanHunters c8ba404646 fix(platform): guard migration 54 against RFC-1123-invalid pool names
The 53-char overflow branch exists because exiting the migration deadlocks the
platform pre-upgrade hook for every tenant. The equivalent deadlock via name
validity was unguarded: the old nodeGroups map key was never schema-constrained
(map[string]NodeGroup, no propertyNames pattern), so a stored parent HR can
carry a key like 'My_Pool' whose own render already failed at apply time. The
migration then built child_name=kubernetes-nodes-<cluster>-My_Pool, kubectl
apply rejected the RFC-1123-invalid HelmRelease name, the script exited 1 under
set -e, and every subsequent platform upgrade re-ran and re-failed.

Validate child_name against the RFC-1123 label pattern alongside the length
check and route failures through the same warn + pin + skip path. Factor the
pin-and-skip logic shared by both guards into pin_pool_objects. Cover the new
branch in migration-54-adopt-path.bats (the fake kubectl apply now rejects an
invalid metadata.name, so the case fails without the guard).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-22 18:34:28 +03:00
Ivan Okhotnikov 386053ba8f fix(platform): harden migration 52 (name overflow, images, adoption tests)
Address review on #3315:
- Name-length overflow no longer exits 1 (which would deadlock the platform
  pre-upgrade hook for every tenant). Instead pin the pool's worker objects
  with resource-policy=keep, skip the child HR, warn loudly, and continue.
- Copy only images.kubectl into the child values, not the parent's whole
  images map (waitForKubeconfig/talosCsrSigner are undeclared in the schema).
- Note that the pre-upgrade window is not quiet (a parent HR edit mid-upgrade
  renders the old chart and self-heals once the new artifact lands).
- Cover the adoption path directly: annotate keep + release-name, idempotent
  skip, unexpected-owner refusal, and the 6-hex KMT anchor rejecting a sibling
  pool's template.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-21 22:57:22 +03:00
Ivan Okhotnikov 9d9068163d fix(platform): copy version/talos/images into adopted pool HR values
The adoption jq used a filter-parameter helper `def pick(o; ks)`; inside
the `reduce` the `.` context is the accumulator, so `o | has($k)` tested
the accumulator, not the source object, and `pick(.; ["version","talos",
"images"])` deterministically returned {}. The child HelmRelease was
therefore created without the tenant's talos/version, so Helm rendered
the chart defaults; since talos.version feeds the KubevirtMachineTemplate
content-hash, the KMT name changed and the MachineDeployment rolled every
live worker VM on upgrade — the exact churn this migration must avoid.
Caught by e2e on dev3 (reproduced on 2/2 real clusters). Switch pick to
a $-parameter so it is evaluated once at the call site.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-21 22:57:06 +03:00
Ivan Okhotnikov a8286a3ac0 fix(platform): harden worker-pool adoption migration
Anchor the KubevirtMachineTemplate sweep to the 6-hex content-hash
suffix so a pool name that is a prefix of a sibling (md0 vs md0-large)
no longer mis-adopts the sibling's KMT and deadlocks the migration.
Adopt the worker objects before creating the child HelmRelease so
helm-controller cannot start the child install into un-adopted objects
and fail Helm ownership validation. Emit the same install/upgrade
strategy (RetryOnFailure) and maxHistory cozystack-api sets, add a
pre-flight release-name length check, and document the rollback path.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-21 22:56:43 +03:00
Ivan Okhotnikov 86ecedb3c0 feat(platform): adopt worker pools into kubernetes-nodes releases (migration 52)
Pre-upgrade migration for the Phase 2 kubernetes chart split. For every
parent kubernetes-<cluster> HelmRelease it creates one child
kubernetes-nodes-<cluster>-<pool> HelmRelease and re-annotates the
existing worker objects onto it with helm.sh/resource-policy=keep, so
the now control-plane-only parent upgrade does not prune the tenant's
running workers. Byte-identical child render (PR-2a golden parity) keeps
the KubevirtMachineTemplate content-hash stable, so no worker VM churns.
Idempotent and strict (exit 1 on any failure). Bumps targetVersion to 53
(contiguous with main; rebase to 53/54 if another 52 lands first).

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
2026-07-21 22:52:05 +03:00
Myasnikov Daniil 47e24f01d1 fix(platform): hand seaweedfs-db over for every instance name, not just the default
Migration 43 performs the v1.5.0 db-split hand-over: it re-owns Cluster/seaweedfs-db
from <name>-system to the new <name>-db release and stamps
helm.sh/resource-policy: keep, so the <name>-system upgrade — whose post-split
chart no longer renders the Cluster — does not prune it as a removed resource.

It compared the owning release name against the literal "seaweedfs-system".
SeaweedFS is a user-creatable kind, so an instance named `foo` is owned by
`foo-system` and was silently skipped: no re-own, no keep. The <name>-system
upgrade then deleted the Cluster, CNPG took its PVC with it, and the filer
metadata — the index for every object in that tenant's S3 — was destroyed. Data
loss, not an outage: the volume PVCs still hold the bytes, nothing can find them.

Verified live on the upgrade stand, which is a natural control group. All five
tenants sit in the SAME failed-upgrade loop (<name>-system last-deployed is
rev 1, a pre-split revision whose manifest still contains the Cluster, so every
retry recomputes the same deletion):

  tenant-root/dsplit/l   instance `seaweedfs`  owner=seaweedfs-db  keep   Cluster alive
  tenant-named           instance `foo`        no Cluster at all          Cluster GONE

The only difference is that migration 43 fired for the default-named instances
and skipped `foo`. That also answers whether `keep` is sufficient: three tenants
survive the identical prune loop with it.

Correction to the reported mechanism: this is the 1.4->1.5 db split (PR #2601,
v1.5.0), not a 1.5->1.6 change — v1.5.3 and main are identical on every
db-split file. And the split does protect the object; it just only protected the
instances someone happened to name `seaweedfs`.

- lib/seaweedfs-db-adopt.sh: shared hand-over matching the `-system` SUFFIX, so
  `foo-system` -> `foo-db` exactly as `seaweedfs-system` -> `seaweedfs-db`.
  Idempotent, and refuses a release named literally `-system` rather than
  annotating an owner of `-db` that no release would claim.
- Migration 43 sources it: covers clusters upgrading from before 43 (1.4.x).
- Migration 53 (new) re-runs it: migrations never re-run, so any cluster already
  at >= 44 ran the hardcoded version and is still exposed. As a pre-upgrade hook
  it lands before <name>-system re-renders, closing the window on THIS upgrade.
  This is the one that matters for 1.5.x -> 1.6.
- targetVersion 53 -> 54.

Tests fail on unfixed code: the non-default-name case produces no ANNOTATE at
all against the hardcoded 43, while the default-name case passes — which is
precisely the bug's shape. They drive the real migration scripts against a fake
kubectl, following hack/migration-50-etcd-adopt.bats.

LIMIT, stated plainly: a Cluster that is already deleted cannot be recovered by
either migration — there is nothing left to re-annotate and the PVC went with
it. Such a tenant needs seaweedfs-db restored from a backup. Runbook Step 0 adds
the ownership audit that distinguishes at-risk from already-lost, and records
that <name>-db can report Ready while its Cluster is gone (it rendered fine; a
later <name>-system prune removed it and Flux has not re-checked) — so the
audit trusts `kubectl get cluster`, not the HelmRelease status.

Review round 2 — two ways this still lost the database:

- OWNERSHIP IS NOT SAFETY. The helper skipped any Cluster already owned by
  <name>-db, treating ownership as proof the hand-over was done. It is not: where
  the hand-over was skipped, <name>-system prunes the Cluster and <name>-db
  RECREATES it under its own ownership with NO keep, while <name>-system's prune
  baseline still lists it — so the next reconcile deletes it again. That is the
  observed delete/recreate loop, and the skip walked straight past it. Verified on
  the stand: tenant-l and tenant-root are seaweedfs-db-owned AND their
  <name>-system deployed revision (rev 1) still contains the Cluster, so only keep
  saves them; tenant-fresh is seaweedfs-db-owned without keep but was installed
  after the split, so its baseline never had one. Telling those apart needs the
  release's deployed manifest, which this script cannot read cheaply or reliably.
  The costs are asymmetric — a needless keep leaves an orphan on delete (now
  reclaimed by the cleanup hook), a missing one loses the database — so keep is
  stamped on every Cluster owned by either side of the split.

- FAIL OPEN, THEN STAMP. `set -euo pipefail` does not abort on a failing command
  substitution in a `for` word-list: the loop ran zero times, the script continued,
  and stamp_cozystack_version ran anyway. Migrations never re-run, so one throttle,
  RBAC hiccup, or not-yet-established CNPG apiservice behind the pre-upgrade hook
  permanently left every at-risk tenant exposed with no later migration to catch
  them. Reproduced: against the old helper a failing fleet scan still logged
  STAMP 54. Now every kubectl failure is fatal EXCEPT the two that genuinely mean
  "nothing to do" — the CNPG resource type not being served (the fail-open that IS
  load-bearing: a cluster without CNPG must still upgrade) and a Cluster vanishing
  between scan and read. An unreadable annotation is no longer indistinguishable
  from an absent one, and the unowned case now warns instead of passing silently.

- ORPHAN ON DELETE. `keep` is permanent (removal deferred to the 1.7 batch
  migrations) and also survives the <name>-db release's own uninstall, so
  `kubectl delete seaweedfs <name>` left the CNPG Cluster, its Postgres pods and
  its PVCs behind — the cleanup hook selects
  app.kubernetes.io/instance=<name>-system, which the db PVCs
  (cnpg.io/cluster=seaweedfs-db) do not carry. Migration 53 widened that from
  pre-1.5 default-named tenants to every instance name, so the hook now deletes
  the Cluster explicitly; its PVCs carry ownerReferences to it and follow.

- Tests assert the stamped VERSION (44 / 54), not a bare "STAMP": a wrong number
  would loop run-migrations.sh forever and the old assertion could not tell.

Review round 3 — the reclaim introduced a data-loss path of its own:

- DELETING ONE APP DELETED ANOTHER APP'S DATABASE. The db chart hardcodes the
  Cluster name `seaweedfs-db` for every instance, and the reclaim was scoped by
  that name in the namespace rather than by what the release owns.
  templates/seaweedfs-db.yaml gates the <name>-db HelmRelease on
  `topology != "Client"`; the cleanup hook had no such gate. Verified by render:
  `helm template s3-remote . --set topology=Client` emits ZERO s3-remote-db
  HelmReleases yet still emitted the Role granting delete on clusters/seaweedfs-db
  AND `kubectl delete cluster.postgresql.cnpg.io -n tenant-root seaweedfs-db`. So
  in a namespace running a server instance plus a Client, deleting the CLIENT
  destroyed the SERVER's filer metadata — CNPG takes the PVC via ownerReferences —
  for an app the operator never touched. Reachable the same way via a second Simple
  instance, whose <name>-db loses Helm's ownership check but which still installs
  and still carries the hook. main's hook was instance-scoped; this was new, and
  mine.
  Now gated exactly as seaweedfs-db.yaml is, and the Job re-checks
  meta.helm.sh/release-name == <release>-db at run time before deleting. The gate
  stops the permission being granted where it can never be legitimate; the runtime
  check stops the delete where it can.

- `|| echo` turned any API/authz/admission failure into exit zero, so backoffLimit
  never retried and hook-delete-policy removed the only cleanup Job while the
  keep-protected Cluster stayed orphaned. A failed reclaim now fails the Job.

- TEST SHELL. The migrations run under /bin/sh = busybox ash (the image is FROM
  alpine, run-migrations.sh is #!/bin/sh), but the tests drove them with bash — so
  the fail-closed guarantees were asserted in a shell that never runs them, which
  matters more than usual given the bug they cover was a fail-open. The scripts are
  now invoked via `sh` and the fake kubectl is POSIX sh. Verified directly against
  the production base image (busybox 1.37.0 / alpine:3.24): `set -o pipefail` is
  supported, every script passes `ash -n`, and the failing-fleet-scan path exits
  non-zero without stamping under real ash. The recipe is in the bats header.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 15:40:48 +05:00
Myasnikov Daniil efa85a4f61 feat(platform): make the etcd-adoption skip-backup hatch reachable
Migration 50 has honoured ETCD_ADOPT_SKIP_BACKUP since it was written, and
its failure paths tell operators to "re-run with ETCD_ADOPT_SKIP_BACKUP=1".
But migration-hook.yaml passes only NAMESPACE/CURRENT_VERSION/
TARGET_VERSION, and the Job is a Helm hook the chart re-renders, so there
has never been a way to set it. The script advertised an escape the
platform made impossible. The endpoint-classification work landing
alongside this makes that worse: it adds refuse-rather-than-guess paths,
so more clusters can now legitimately stop here.

Plumb it through migrations.etcdAdoptSkipBackup, default false. Skipping
the pre-adoption snapshot rewires ownership of LIVE etcd storage with no
way back, so it stays opt-in and deliberate — the flag is the last resort
for a cluster that has nothing to fix, not the first response to a
refusal.

Render "1"/"0" rather than the bare bool. The script matches the literal
string "1", so a bool would render "true", sail through the Job spec
looking correct, and be ignored — a safety valve that is set but silently
disregarded is worse than one that was never offered, because the operator
believes they opted in. The value is normalised through a string, so a
bool, a quoted "true" out of the Package CR's Values JSON, or a numeric 1
all land on the same answer, while nil or garbage resolves to "0": taking
this hatch must require an affirmative spelling, never a typo. Resolved
with `dig` so a Platform Package predating the key renders "0" instead of
failing the render of the entire platform chart and bricking every
upgrade.

Widen the script's own check to accept 1/true/yes as well. The template
guarantees the GitOps path, but an operator running this image by hand
will reasonably type "true", and that is exactly the moment they are
already stuck. The two layers cover independent failure modes.

Emit the var unconditionally, including the "0" default, so the rendered
Job states the cluster's safety posture outright rather than leaving an
operator to diff values to find out whether a snapshot was skipped.

Point the failure messages at the values path that actually exists, and
describe it honestly: editing the Platform Package and letting Flux
re-render is a real operation, not a one-liner, and the flag has to be
reverted afterwards or the next etcd migration skips its snapshot too.
Fixing the condition the migration names remains the recommended path —
the migration is idempotent, so re-running after a fix is safe.

templates/migration-hook.yaml had no coverage at all, because the Job only
renders when a lookup of the cozystack-version ConfigMap reports a version
below targetVersion, and lookup is nil under helm template. The new suite
mocks that ConfigMap and asserts the RENDERED VALUE — enabled, disabled,
key absent, stringified true, garbage, and the pre-existing three env vars
— rather than merely that the variable exists, which is what let the trap
above go unnoticed in the first place.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-20 11:29:09 +05:00