fix(seaweedfs): retry the data-plane release instead of rolling it back

The admin volume and the two lifecycle pods are new demands on tenant
quota, so an upgrade into a namespace with no headroom can wait out its
timeout. Flux's default remediation then rolls the release back, and
with unlimited retries it does so on every attempt: each rollback drops
the [grpc.s3] section and the s3 restart annotation, each retry brings
them back, and the s3 pods restart twice per cycle until quota frees up.

RetryOnFailure keeps what was applied and retries it, as the parent
release that the apiserver builds for the application already does.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
This commit is contained in:
Alexey Artamonov committed 2026-11-05 21:03:43 +03:00
1 parent 27e0aa2880
commit a70a1cf83b
2 files changed
+33

No files matched your search

@@ -214,10 +214,21 @@ spec:
namespace: cozy-system
interval: 30s
timeout: 10m
{{- /* Flux remediates a failed install by uninstalling and a failed upgrade
by rolling back, and with retries: -1 it does so on every attempt. A
release waiting on tenant quota for the admin volume would then roll
the s3 pods back and forth with the [grpc.s3] restart annotation.
RetryOnFailure keeps what was applied and retries it. */}}
install:
strategy:
name: RetryOnFailure
retryInterval: 30s
remediation:
retries: -1
upgrade:
strategy:
name: RetryOnFailure
retryInterval: 30s
remediation:
retries: -1
valuesFrom:
@@ -194,3 +194,25 @@ tests:
asserts:
- hasDocuments:
count: 0
# The admin PVC and the two lifecycle pods can leave an upgrade waiting on a
# full tenant quota. Under Flux's default rollback remediation each failed
# attempt would roll back the [grpc.s3] section and its restart annotation
# and the next one would bring them back, restarting the s3 pods on every
# cycle. Retrying in place keeps what was applied while the release waits.
- it: retries the data-plane release in place instead of rolling it back
template: templates/seaweedfs.yaml
set: *platform
documentSelector:
path: metadata.name
value: seaweedfs-system
asserts:
- equal:
path: spec.install.strategy.name
value: RetryOnFailure
- equal:
path: spec.upgrade.strategy.name
value: RetryOnFailure
- equal:
path: spec.upgrade.remediation.retries
value: -1