Files
cozystack/packages
Aleksei Sviridkin 6d403c3015 fix(tenant): retry failed etcd and seaweedfs installs in place
The tenant chart renders both releases with Flux's default install
strategy, and install remediation is an uninstall in every case Flux
offers. Neither release disables hooks on it, so a remediated install
runs each chart's post-delete cleanup Job, and those Jobs delete data on
purpose. The etcd one removes the data-etcd-<ordinal> PVCs by name,
because the etcd-operator does not reclaim them when the EtcdCluster
goes. The seaweedfs one removes the volume PVCs by label and explicitly
reclaims the keep-protected CNPG Cluster holding the filer metadata,
precisely because nothing else removes it. Both are correct for a real
uninstall. Remediation is what calls them, on a release that is only
starting slowly.

So an install that misses its budget does not merely restart. It
destroys the datastore the tenant control planes keep their state in, or
the object store's data together with the metadata database that indexes
it, and the next attempt begins from nothing.

RetryOnFailure keeps the manifests applied and retries the failed
release, so no uninstall happens and neither cleanup Job runs. Both
actions carry the strategy because the controller performs that retry as
an upgrade, so on install alone the retry's own failure falls through to
upgrade remediation. The cost is that a real upgrade failure retries
rather than rolls back, which gives up no stable safety net: with
retries: -1 and the default strategy, upgrade remediation already rolled
back and retried without end. The remediation blocks stay because an
absent release consults install and an out-of-sync one consults upgrade,
both by retry count rather than by strategy.

The other side of the trade is that the uninstall was also the only
automatic clean slate. On a tenant holding no data yet, an install that
wedges on a bad resource state now retries against that state until an
operator removes the release. For a release that owns data this is the
right exchange, because lost data does not come back and a wedged
install does.

This covers two of the tenant's data-owning releases. The chart's
monitoring release destroys its storage on a remediated install as well
and is outside this change, as is the rest of the catalog, whose nested
releases want the question asked one at a time.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-08 02:26:31 +03:00
..