fix(backupstrategy-controller): repair lookup-gated backup objects

The default Strategy CRs and the Velero BackupStorageLocation are
Helm-templated behind a `lookup` of the BucketClaim the same chart
creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of
the COSI Secret. When those lookups are empty at install time the objects
were silently skipped, and helm-controller does not re-render a release
whose chart and values did not change (drift detection is off on
operator-generated HelmReleases), so the skip was permanent: clusters ran
for months with only BackupClass cozy-default and no Strategy CRs at all.
That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd
migration, which reads the projected credentials Secret.

Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the
bucket name is resolvable from the projector's source Secret, it checks
that every object cozy-default routes to exists and forces one real Helm
upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing.
Helm remains the objects' only author; the gate only makes sure the render
that produces them actually happens. It cannot be a render-time `fail` in
this chart: the chart is the producer of the Bucket its own lookup reads,
so a failed render would deadlock the condition.

The bucket chart's user-credentials template CAN fail safely — the
BucketAccess that produces the COSI Secret belongs to the parent release
and that release retries forever — so make it fail loudly instead of
skipping, with requireUserCredentials=false as the offline-render escape.

Also correct the chart and doc comments that claimed Flux re-renders on
its interval, and document the two-release manual recovery for clusters
already affected.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>

cmd/backupstrategy-controller/main.go

@@ -27,6 +27,7 @@ import (
 
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/types"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
 	ctrl "sigs.k8s.io/controller-runtime"
@@ -203,6 +204,29 @@ func main() {
 		}
 	}
 
+	// The default Strategy CRs and the Velero BSL are Helm-templated behind
+	// a lookup of the BucketClaim the same chart creates, so a fresh install
+	// renders them empty — permanently, because helm-controller does not
+	// re-render an unchanged, successful release. The gate detects that and
+	// forces the one real Helm upgrade that materialises them. Disabled when
+	// the HelmRelease coordinates are not plumbed (e.g. a chart-less local run).
+	if hrName := os.Getenv("BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME"); hrName != "" {
+		gate := &backupcontroller.DefaultObjectsGate{
+			Client:          mgr.GetClient(),
+			Config:          credentialsConfig,
+			BackupClassName: os.Getenv("BACKUP_DEFAULT_OBJECTS_BACKUPCLASS"),
+			HelmRelease: types.NamespacedName{
+				Namespace: os.Getenv("BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAMESPACE"),
+				Name:      hrName,
+			},
+			VeleroNamespace: os.Getenv("BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE"),
+		}
+		if err := gate.SetupWithManager(mgr); err != nil {
+			setupLog.Error(err, "unable to add DefaultObjectsGate runnable")
+			os.Exit(1)
+		}
+	}
+
 	if err = (&backupcontroller.BackupJobReconciler{
 		Client:            mgr.GetClient(),
 		Scheme:            mgr.GetScheme(),

docs/operations/backup-classes.md

@@ -93,9 +93,46 @@ On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is
 
 ### Cozy-default Bucket bootstrap
 
-`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a short reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. The HelmRelease re-reconciles on its interval (5 minutes by default — set by the cozystack operator's `helmrelease-interval` flag, not a Flux default), at which point the populated BucketClaim status causes the missing strategy templates to materialise.
+`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name (`bucket-<claim-UID>`, so it cannot be computed in advance) and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster.
 
-If you need the BackupClass functional immediately (e.g. an e2e), trigger a Flux reconcile (`flux reconcile helmrelease backupstrategy-controller -n cozy-backup-controller`) once you see `kubectl get bucketclaim -n tenant-root bucket-cozy-backups -o jsonpath='{.status.bucketName}'` non-empty.
+**That skip does not repair itself on a reconcile.** helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps `BackupClass cozy-default` with **no** `Strategy` CRs and **no** Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration.
+
+Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default). Once the bucket name is resolvable it checks that every object `cozy-default` routes to exists, and stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the `backupstrategy-controller` HelmRelease to force the real Helm upgrade that re-runs the lookups. Expect the objects within one minute of the bucket becoming ready. Watch it with:
+
+```bash
+kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate
+```
+
+#### Manual recovery on an affected cluster
+
+Only needed on a cluster running a version without the gate (or with `reconcileDefaultObjects: false`). A plain `flux reconcile helmrelease` does **nothing** here — it does not re-render. You need a forced upgrade, and **both** annotations: `forceAt` is what makes helm-controller run a real Helm upgrade, and it is only honoured together with `requestedAt`.
+
+First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup:
+
+```bash
+kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}'
+```
+
+Then force the two releases, **in this order**:
+
+```bash
+# 1. The credentials Secret. It is rendered by the -system release, not by
+#    bucket-cozy-backups — easy to miss, and the projector (hence every
+#    strategy and Velero) has no source without it.
+ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \
+  reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
+kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials
+
+# 2. The Strategy CRs and the Velero BSL.
+ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
+kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \
+  reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite
+kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null
+kubectl -n cozy-velero get backupstoragelocation cozy-default
+```
+
+The race is nested: step 2's `lookup` resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both.
 
 ### Observability
 
@@ -106,6 +143,11 @@ The credentials projector emits two Prometheus counters labelled by `namespace`
 
 Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping.
 
+The default-objects gate emits two more:
+
+- `cozystack_backup_default_objects_missing{backupclass="cozy-default"}` — how many objects `cozy-default` routes to are absent. **This is the alert that would have caught the missing Strategy CRs**: it is non-zero whatever the HelmRelease's `Ready` condition says. Alert on `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0`.
+- `cozystack_backup_default_objects_force_reconciles_total` — forced Helm upgrades issued. A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race.
+
 ## Admin overrides for `cozy-default`
 
 `cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-foundationdb`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR:

internal/backupcontroller/default_objects_gate.go

@@ -0,0 +1,356 @@
+package backupcontroller
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"sort"
+	"time"
+
+	"github.com/go-logr/logr"
+	"github.com/prometheus/client_golang/prometheus"
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/dynamic"
+	"k8s.io/client-go/rest"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/apiutil"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/controller-runtime/pkg/metrics"
+
+	backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
+)
+
+// defaultObjectsMissing reports how many of the objects the platform
+// BackupClass depends on are absent from the cluster. It is the signal an
+// operator alerts on: a non-zero value that does not return to zero means
+// the platform default backups are not usable, whatever the HelmRelease's
+// Ready condition says.
+var defaultObjectsMissing = prometheus.NewGaugeVec(
+	prometheus.GaugeOpts{
+		Name: "cozystack_backup_default_objects_missing",
+		Help: "Number of objects referenced by the platform BackupClass that do not exist in the cluster.",
+	},
+	[]string{"backupclass"},
+)
+
+// defaultObjectsForceReconciles counts the forced Helm upgrades the gate
+// issued to materialise those objects. A counter that keeps climbing means
+// the forced render is not producing the objects (e.g. a CRD is missing),
+// which is a different failure than the install-time race this gate closes.
+var defaultObjectsForceReconciles = prometheus.NewCounterVec(
+	prometheus.CounterOpts{
+		Name: "cozystack_backup_default_objects_force_reconciles_total",
+		Help: "Number of forced HelmRelease reconciles issued to create missing platform backup objects.",
+	},
+	[]string{"namespace", "name"},
+)
+
+func init() {
+	metrics.Registry.MustRegister(defaultObjectsMissing, defaultObjectsForceReconciles)
+}
+
+const (
+	// strategyAPIGroup is the API group of the strategy.backups.cozystack.io
+	// CRs a BackupClass routes to. BackupClassStrategy.StrategyRef is a
+	// TypedLocalObjectReference and carries no version, so the group+kind
+	// are mapped to a resource through the RESTMapper.
+	strategyAPIGroup = "strategy.backups.cozystack.io"
+
+	// defaultBackupStorageLocationName is the Velero BSL the cozy-default
+	// Velero strategies reference via storageLocation. It is rendered by the
+	// same lookup-gated template as the Strategy CRs, so it belongs to the
+	// same existence check.
+	defaultBackupStorageLocationName = "cozy-default"
+)
+
+var backupStorageLocationGVR = schema.GroupVersionResource{
+	Group:    "velero.io",
+	Version:  "v1",
+	Resource: "backupstoragelocations",
+}
+
+// DefaultObjectsGate closes a race the chart's templates cannot close by
+// themselves.
+//
+// The default Strategy CRs and the Velero BackupStorageLocation are
+// Helm-templated behind a `lookup` of the BucketClaim the SAME chart
+// creates, because the S3 bucket name is assigned by the COSI driver
+// (`bucket-<claim-UID>`) and is therefore unknowable at render time. On a
+// fresh install the lookup is empty and those templates render nothing.
+// helm-controller does not re-render a release that succeeded and whose
+// chart and values did not change (drift detection is off on
+// operator-generated HelmReleases), so the skip is PERMANENT: the objects
+// are never created, and the only recovery is a forced Helm upgrade
+// (reconcile.fluxcd.io/forceAt + requestedAt — a plain reconcile request
+// does not re-render).
+//
+// This runnable performs that recovery automatically. Once the bucket name
+// is resolvable — read from the same platform credentials Secret the
+// projector already consumes, so no new dependency and no `lookup` — it
+// checks that every object the platform BackupClass routes to actually
+// exists, and forces one Helm upgrade when any is missing. A forced
+// upgrade re-runs the lookups, so the gated templates materialise.
+//
+// It deliberately does not create the objects itself: their bodies are
+// values-driven (endpoint, region, forcePathStyle, the Altinity strategy's
+// whole PodTemplateSpec) and Helm remains their single owner. The gate only
+// makes sure the render that produces them actually happens.
+type DefaultObjectsGate struct {
+	// Client reads the BackupClass (cached) and the source credentials
+	// Secret (uncached — the manager disables the Secret cache).
+	Client client.Client
+	// Interface is a dynamic client used for the per-object existence
+	// checks and the HelmRelease annotation patch. Going through the
+	// dynamic client keeps these reads off the controller-runtime cache,
+	// so they need only `get` RBAC and start no cluster-wide informers.
+	dynamic.Interface
+	meta.RESTMapper
+
+	// Config supplies the source credentials Secret coordinates. Reused
+	// verbatim from the credentials projector.
+	Config BackupCredentialsConfig
+	// BackupClassName is the platform BackupClass whose strategyRefs
+	// enumerate the objects that must exist. Empty disables the gate.
+	BackupClassName string
+	// HelmRelease is the release that renders those objects and is forced
+	// when they are missing. Empty name disables the gate.
+	HelmRelease types.NamespacedName
+	// VeleroNamespace is where the cozy-default BackupStorageLocation
+	// lives. Empty skips the BSL check (velero.bslEnabled=false).
+	VeleroNamespace string
+
+	// Period is the check interval. Defaults to 1 minute.
+	Period time.Duration
+	// MinForceInterval throttles forced upgrades so a condition the forced
+	// render cannot fix (missing CRD, wrong bucket) does not turn into a
+	// hot loop against helm-controller. Defaults to 5 minutes.
+	MinForceInterval time.Duration
+
+	// now is a test seam for the throttle clock.
+	now       func() time.Time
+	lastForce time.Time
+}
+
+var (
+	_ manager.Runnable               = (*DefaultObjectsGate)(nil)
+	_ manager.LeaderElectionRunnable = (*DefaultObjectsGate)(nil)
+)
+
+// NeedLeaderElection keeps a single replica forcing the release. Two
+// replicas racing the same annotation patch would double the Helm upgrades
+// for no benefit.
+func (g *DefaultObjectsGate) NeedLeaderElection() bool { return true }
+
+// SetupWithManager wires the dynamic client and the RESTMapper from the
+// manager's rest.Config, mirroring RestoreJobReconciler, and registers the
+// gate as a manager Runnable.
+func (g *DefaultObjectsGate) SetupWithManager(mgr ctrl.Manager) error {
+	cfg := mgr.GetConfig()
+	var err error
+	if g.Interface, err = dynamic.NewForConfig(cfg); err != nil {
+		return err
+	}
+	var h *http.Client
+	if h, err = rest.HTTPClientFor(cfg); err != nil {
+		return err
+	}
+	if g.RESTMapper, err = apiutil.NewDynamicRESTMapper(cfg, h); err != nil {
+		return err
+	}
+	return mgr.Add(g)
+}
+
+func (g *DefaultObjectsGate) Start(ctx context.Context) error {
+	logger := log.FromContext(ctx).WithName("default-objects-gate")
+	if g.BackupClassName == "" || g.HelmRelease.Name == "" || g.HelmRelease.Namespace == "" || !g.Config.IsEnabled() {
+		logger.V(1).Info("default backup objects gate disabled",
+			"backupClass", g.BackupClassName,
+			"helmRelease", g.HelmRelease.String(),
+			"credentialsConfigured", g.Config.IsEnabled())
+		return nil
+	}
+	if g.Period == 0 {
+		g.Period = time.Minute
+	}
+	if g.MinForceInterval == 0 {
+		g.MinForceInterval = 5 * time.Minute
+	}
+	if g.now == nil {
+		g.now = time.Now
+	}
+
+	tick := time.NewTicker(g.Period)
+	defer tick.Stop()
+	g.checkAndLog(ctx, logger)
+	for {
+		select {
+		case <-ctx.Done():
+			return nil
+		case <-tick.C:
+			g.checkAndLog(ctx, logger)
+		}
+	}
+}
+
+func (g *DefaultObjectsGate) checkAndLog(ctx context.Context, logger logr.Logger) {
+	missing, forced, err := g.Check(ctx)
+	switch {
+	case err != nil:
+		// Info, not Error: every branch here is either transient (the
+		// bucket is still being provisioned) or already exposed as a
+		// metric, and this runs every minute for the life of the cluster.
+		logger.Info("default backup objects check failed", "error", err.Error())
+	case len(missing) > 0 && forced:
+		logger.Info("forced a Helm upgrade to create missing default backup objects",
+			"helmRelease", g.HelmRelease.String(), "missing", missing)
+	case len(missing) > 0:
+		logger.Info("default backup objects still missing, force throttled",
+			"helmRelease", g.HelmRelease.String(), "missing", missing,
+			"minForceInterval", g.MinForceInterval.String())
+	default:
+		logger.V(1).Info("all default backup objects present")
+	}
+}
+
+// Check resolves the platform bucket name, verifies that every object the
+// BackupClass routes to exists, and forces one Helm upgrade when any is
+// missing. It returns the missing objects (formatted for logs) and whether
+// a forced upgrade was issued on this call.
+//
+// It is a no-op while the bucket name is unresolvable: forcing then would
+// re-render the same empty lookup. The bucket name is read from the
+// projector's source Secret rather than from the BucketClaim, so the gate
+// needs no objectstorage RBAC and works for the external-S3 path too.
+func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) {
+	src := &corev1.Secret{}
+	if err := g.Client.Get(ctx, types.NamespacedName{Namespace: g.Config.SourceNamespace, Name: g.Config.SourceSecretName}, src); err != nil {
+		return nil, false, fmt.Errorf("get source credentials Secret %s/%s: %w", g.Config.SourceNamespace, g.Config.SourceSecretName, err)
+	}
+	creds, err := parseSourceSecret(src)
+	if err != nil {
+		return nil, false, err
+	}
+	if creds.bucket == "" {
+		// The bucket is not provisioned yet (or an admin-managed Secret
+		// omits the name). Nothing a re-render could resolve.
+		return nil, false, nil
+	}
+
+	backupClass := &backupsv1alpha1.BackupClass{}
+	if err := g.Client.Get(ctx, client.ObjectKey{Name: g.BackupClassName}, backupClass); err != nil {
+		return nil, false, fmt.Errorf("get BackupClass %s: %w", g.BackupClassName, err)
+	}
+
+	missing, err := g.missingObjects(ctx, backupClass)
+	if err != nil {
+		return nil, false, err
+	}
+	defaultObjectsMissing.WithLabelValues(g.BackupClassName).Set(float64(len(missing)))
+	if len(missing) == 0 {
+		return nil, false, nil
+	}
+
+	now := g.now()
+	if !g.lastForce.IsZero() && now.Sub(g.lastForce) < g.MinForceInterval {
+		return missing, false, nil
+	}
+	if err := g.forceHelmRelease(ctx, now); err != nil {
+		if apierrors.IsNotFound(err) {
+			// Not a Flux-managed install (a plain `helm install` for local
+			// development). There is nothing to force; stay quiet rather
+			// than logging an error every tick.
+			g.lastForce = now
+			return missing, false, nil
+		}
+		return missing, false, fmt.Errorf("force HelmRelease %s: %w", g.HelmRelease.String(), err)
+	}
+	g.lastForce = now
+	defaultObjectsForceReconciles.WithLabelValues(g.HelmRelease.Namespace, g.HelmRelease.Name).Inc()
+	return missing, true, nil
+}
+
+// missingObjects returns a sorted, de-duplicated list of the
+// "<kind>/<name>" objects the BackupClass references that do not exist,
+// plus the Velero BSL when a Velero namespace is configured.
+//
+// An unmappable group/kind (the CRD is not installed) is NOT reported as
+// missing: no Helm re-render can create an object whose CRD is absent, and
+// counting it would keep the gate forcing upgrades forever.
+func (g *DefaultObjectsGate) missingObjects(ctx context.Context, backupClass *backupsv1alpha1.BackupClass) ([]string, error) {
+	seen := map[string]struct{}{}
+	var missing []string
+
+	for _, strategy := range backupClass.Spec.Strategies {
+		name := strategy.StrategyRef.Name
+		kind := strategy.StrategyRef.Kind
+		if name == "" || kind == "" {
+			continue
+		}
+		group := strategyAPIGroup
+		if strategy.StrategyRef.APIGroup != nil && *strategy.StrategyRef.APIGroup != "" {
+			group = *strategy.StrategyRef.APIGroup
+		}
+		key := fmt.Sprintf("%s/%s", kind, name)
+		if _, dup := seen[key]; dup {
+			continue
+		}
+		seen[key] = struct{}{}
+
+		mapping, err := g.RESTMapping(schema.GroupKind{Group: group, Kind: kind})
+		if err != nil {
+			if meta.IsNoMatchError(err) {
+				continue
+			}
+			return nil, fmt.Errorf("map %s/%s: %w", group, kind, err)
+		}
+		// Strategy CRs are cluster-scoped; Namespace("") is correct for
+		// both scopes on the dynamic client.
+		_, err = g.Resource(mapping.Resource).Get(ctx, name, metav1.GetOptions{})
+		switch {
+		case err == nil:
+		case apierrors.IsNotFound(err):
+			missing = append(missing, key)
+		default:
+			return nil, fmt.Errorf("get %s %s: %w", kind, name, err)
+		}
+	}
+
+	if g.VeleroNamespace != "" {
+		_, err := g.Resource(backupStorageLocationGVR).Namespace(g.VeleroNamespace).Get(ctx, defaultBackupStorageLocationName, metav1.GetOptions{})
+		switch {
+		case err == nil:
+		case apierrors.IsNotFound(err):
+			missing = append(missing, fmt.Sprintf("BackupStorageLocation/%s", defaultBackupStorageLocationName))
+		case meta.IsNoMatchError(err):
+			// Velero CRDs absent: nothing to materialise.
+		default:
+			return nil, fmt.Errorf("get BackupStorageLocation %s: %w", defaultBackupStorageLocationName, err)
+		}
+	}
+
+	sort.Strings(missing)
+	return missing, nil
+}
+
+// forceHelmRelease stamps BOTH reconcile.fluxcd.io/forceAt and
+// reconcile.fluxcd.io/requestedAt. requestedAt alone only asks
+// helm-controller to reconcile, which is a no-op for a release whose chart
+// and values are unchanged — it is forceAt that makes it run a real Helm
+// upgrade, and forceAt is only honoured together with requestedAt.
+func (g *DefaultObjectsGate) forceHelmRelease(ctx context.Context, now time.Time) error {
+	stamp := now.UTC().Format(time.RFC3339Nano)
+	patch := fmt.Sprintf(
+		`{"metadata":{"annotations":{"reconcile.fluxcd.io/forceAt":%q,"reconcile.fluxcd.io/requestedAt":%q}}}`,
+		stamp, stamp,
+	)
+	_, err := g.Resource(helmReleaseGVR).Namespace(g.HelmRelease.Namespace).
+		Patch(ctx, g.HelmRelease.Name, types.MergePatchType, []byte(patch), metav1.PatchOptions{})
+	return err
+}

internal/backupcontroller/default_objects_gate_test.go

@@ -0,0 +1,416 @@
+package backupcontroller
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/schema"
+	"k8s.io/apimachinery/pkg/types"
+	dynamicfake "k8s.io/client-go/dynamic/fake"
+	k8stesting "k8s.io/client-go/testing"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+
+	backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
+)
+
+// gateGVKs are the kinds the gate touches in these tests: the
+// strategy.backups.cozystack.io CRs cozy-default routes to, the Velero BSL,
+// and the HelmRelease it patches. The RESTMapper and the dynamic fake's
+// resource registry are both derived from this one list so the plural the
+// gate resolves and the plural the fake serves cannot drift apart.
+var gateGVKs = []schema.GroupVersionKind{
+	{Group: strategyAPIGroup, Version: "v1alpha1", Kind: "CNPG"},
+	{Group: strategyAPIGroup, Version: "v1alpha1", Kind: "Etcd"},
+	{Group: strategyAPIGroup, Version: "v1alpha1", Kind: "Velero"},
+	{Group: "velero.io", Version: "v1", Kind: "BackupStorageLocation"},
+	{Group: "helm.toolkit.fluxcd.io", Version: "v2", Kind: "HelmRelease"},
+}
+
+func gateScheme() *runtime.Scheme {
+	scheme := runtime.NewScheme()
+	_ = corev1.AddToScheme(scheme)
+	_ = backupsv1alpha1.AddToScheme(scheme)
+	return scheme
+}
+
+func gateListKinds() map[schema.GroupVersionResource]string {
+	out := map[schema.GroupVersionResource]string{}
+	for _, gvk := range gateGVKs {
+		gvr, _ := meta.UnsafeGuessKindToResource(gvk)
+		out[gvr] = gvk.Kind + "List"
+	}
+	return out
+}
+
+// gateRESTMapper resolves group+kind (the only coordinates a
+// BackupClass strategyRef carries) to a resource, which is what the gate
+// does in production through the dynamic RESTMapper. The group versions must
+// be passed as defaults: RESTMapping is called without a version.
+func gateRESTMapper() meta.RESTMapper {
+	groupVersions := map[schema.GroupVersion]struct{}{}
+	for _, gvk := range gateGVKs {
+		groupVersions[gvk.GroupVersion()] = struct{}{}
+	}
+	var gvs []schema.GroupVersion
+	for gv := range groupVersions {
+		gvs = append(gvs, gv)
+	}
+	m := meta.NewDefaultRESTMapper(gvs)
+	for _, gvk := range gateGVKs {
+		scope := meta.RESTScopeRoot
+		if gvk.Kind == "BackupStorageLocation" || gvk.Kind == "HelmRelease" {
+			scope = meta.RESTScopeNamespace
+		}
+		m.Add(gvk, scope)
+	}
+	return m
+}
+
+func apiGroup(s string) *string { return &s }
+
+// cozyDefaultBackupClass mirrors the routes the chart's
+// backupclass-default.yaml renders unconditionally: it is the manifest of
+// what must exist, which is exactly why the gate reads it instead of
+// hard-coding a list of Strategy names.
+func cozyDefaultBackupClass() *backupsv1alpha1.BackupClass {
+	ref := func(kind, name string) backupsv1alpha1.BackupClassStrategy {
+		return backupsv1alpha1.BackupClassStrategy{
+			Application: backupsv1alpha1.ApplicationSelector{
+				APIGroup: apiGroup("apps.cozystack.io"),
+				Kind:     kind + "App",
+			},
+			StrategyRef: corev1.TypedLocalObjectReference{
+				APIGroup: apiGroup(strategyAPIGroup),
+				Kind:     kind,
+				Name:     name,
+			},
+		}
+	}
+	return &backupsv1alpha1.BackupClass{
+		ObjectMeta: metav1.ObjectMeta{Name: "cozy-default"},
+		Spec: backupsv1alpha1.BackupClassSpec{
+			Strategies: []backupsv1alpha1.BackupClassStrategy{
+				ref("CNPG", "cozy-default-cnpg"),
+				ref("Etcd", "cozy-default-etcd"),
+				ref("Velero", "cozy-default-velero-vminstance"),
+				ref("Velero", "cozy-default-velero-vmdisk"),
+			},
+		},
+	}
+}
+
+func strategyObject(kind, name string) *unstructured.Unstructured {
+	u := &unstructured.Unstructured{}
+	u.SetAPIVersion(strategyAPIGroup + "/v1alpha1")
+	u.SetKind(kind)
+	u.SetName(name)
+	return u
+}
+
+func helmReleaseObject() *unstructured.Unstructured {
+	u := &unstructured.Unstructured{}
+	u.SetAPIVersion("helm.toolkit.fluxcd.io/v2")
+	u.SetKind("HelmRelease")
+	u.SetNamespace("cozy-backup-controller")
+	u.SetName("backupstrategy-controller")
+	return u
+}
+
+func newGate(t *testing.T, ctrlObjs []client.Object, dynObjs ...runtime.Object) (*DefaultObjectsGate, *dynamicfake.FakeDynamicClient) {
+	t.Helper()
+	dyn := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), gateListKinds(), dynObjs...)
+	g := &DefaultObjectsGate{
+		Client: fake.NewClientBuilder().WithScheme(gateScheme()).WithObjects(ctrlObjs...).Build(),
+		Config: BackupCredentialsConfig{
+			SourceNamespace:  "tenant-root",
+			SourceSecretName: "bucket-cozy-backups-system-credentials",
+			TargetSecretName: "cozy-backups-creds",
+		},
+		BackupClassName: "cozy-default",
+		HelmRelease: types.NamespacedName{
+			Namespace: "cozy-backup-controller",
+			Name:      "backupstrategy-controller",
+		},
+		VeleroNamespace:  "cozy-velero",
+		MinForceInterval: 5 * time.Minute,
+		now:              func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) },
+	}
+	g.Interface = dyn
+	g.RESTMapper = gateRESTMapper()
+	return g, dyn
+}
+
+func sourceSecret(bucket string) *corev1.Secret {
+	data := map[string][]byte{
+		"accessKey": []byte("AK"),
+		"secretKey": []byte("SK"),
+		"endpoint":  []byte("s3.example.com"),
+	}
+	if bucket != "" {
+		data["bucketName"] = []byte(bucket)
+	}
+	return &corev1.Secret{
+		ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-root", Name: "bucket-cozy-backups-system-credentials"},
+		Data:       data,
+	}
+}
+
+func bslObject() *unstructured.Unstructured {
+	u := &unstructured.Unstructured{}
+	u.SetAPIVersion("velero.io/v1")
+	u.SetKind("BackupStorageLocation")
+	u.SetNamespace("cozy-velero")
+	u.SetName(defaultBackupStorageLocationName)
+	return u
+}
+
+func forceAnnotations(t *testing.T, dyn *dynamicfake.FakeDynamicClient) map[string]string {
+	t.Helper()
+	hr, err := dyn.Resource(helmReleaseGVR).Namespace("cozy-backup-controller").
+		Get(context.Background(), "backupstrategy-controller", metav1.GetOptions{})
+	if err != nil {
+		t.Fatalf("get HelmRelease: %v", err)
+	}
+	return hr.GetAnnotations()
+}
+
+// TestCheckForcesWhenObjectsMissing is the bug this gate exists for: the
+// bucket name resolves, so a Helm re-render WOULD produce the gated
+// templates, but the objects are absent because the install-time lookup was
+// empty and helm-controller never re-rendered. The gate must force a real
+// Helm upgrade — BOTH annotations, because requestedAt alone is a no-op for
+// an unchanged release.
+func TestCheckForcesWhenObjectsMissing(t *testing.T) {
+	g, dyn := newGate(t,
+		[]client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()},
+		helmReleaseObject(),
+	)
+
+	missing, forced, err := g.Check(context.Background())
+	if err != nil {
+		t.Fatalf("Check: %v", err)
+	}
+	if !forced {
+		t.Fatal("expected a forced Helm upgrade when default objects are missing")
+	}
+	// 4 strategyRefs (Velero twice under different names) + the BSL.
+	if len(missing) != 5 {
+		t.Fatalf("missing = %v, want 5 entries", missing)
+	}
+	ann := forceAnnotations(t, dyn)
+	forceAt, ok := ann["reconcile.fluxcd.io/forceAt"]
+	if !ok || forceAt == "" {
+		t.Errorf("reconcile.fluxcd.io/forceAt not set: %v", ann)
+	}
+	requestedAt, ok := ann["reconcile.fluxcd.io/requestedAt"]
+	if !ok || requestedAt == "" {
+		t.Errorf("reconcile.fluxcd.io/requestedAt not set: %v", ann)
+	}
+	if forceAt != requestedAt {
+		t.Errorf("forceAt (%q) and requestedAt (%q) must carry the same stamp", forceAt, requestedAt)
+	}
+}
+
+// TestCheckNoopWhenAllPresent pins the steady state: no forced upgrades
+// once the objects exist, otherwise the gate would rewrite the annotation
+// forever and re-run a Helm upgrade every MinForceInterval for the life of
+// the cluster.
+func TestCheckNoopWhenAllPresent(t *testing.T) {
+	g, dyn := newGate(t,
+		[]client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()},
+		helmReleaseObject(),
+		strategyObject("CNPG", "cozy-default-cnpg"),
+		strategyObject("Etcd", "cozy-default-etcd"),
+		strategyObject("Velero", "cozy-default-velero-vminstance"),
+		strategyObject("Velero", "cozy-default-velero-vmdisk"),
+		bslObject(),
+	)
+
+	missing, forced, err := g.Check(context.Background())
+	if err != nil {
+		t.Fatalf("Check: %v", err)
+	}
+	if len(missing) != 0 || forced {
+		t.Fatalf("missing = %v, forced = %v, want none", missing, forced)
+	}
+	if ann := forceAnnotations(t, dyn); len(ann) != 0 {
+		t.Errorf("HelmRelease annotated in the steady state: %v", ann)
+	}
+}
+
+// TestCheckSkipsWhileBucketUnresolved pins the precondition. Forcing a
+// re-render before the COSI driver has published a bucket name would just
+// re-run the same empty lookup and burn a Helm upgrade, so the gate waits.
+func TestCheckSkipsWhileBucketUnresolved(t *testing.T) {
+	g, dyn := newGate(t,
+		[]client.Object{sourceSecret(""), cozyDefaultBackupClass()},
+		helmReleaseObject(),
+	)
+
+	missing, forced, err := g.Check(context.Background())
+	if err != nil {
+		t.Fatalf("Check: %v", err)
+	}
+	if len(missing) != 0 || forced {
+		t.Fatalf("missing = %v, forced = %v, want none while the bucket is unresolved", missing, forced)
+	}
+	if ann := forceAnnotations(t, dyn); len(ann) != 0 {
+		t.Errorf("HelmRelease forced before the bucket resolved: %v", ann)
+	}
+}
+
+// TestCheckThrottlesRepeatedForces pins the throttle. A condition a
+// re-render cannot fix (missing CRD, unreachable bucket) must not turn into
+// a hot loop of Helm upgrades against helm-controller.
+func TestCheckThrottlesRepeatedForces(t *testing.T) {
+	g, _ := newGate(t,
+		[]client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()},
+		helmReleaseObject(),
+	)
+	base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
+	g.now = func() time.Time { return base }
+
+	if _, forced, err := g.Check(context.Background()); err != nil || !forced {
+		t.Fatalf("first Check: forced = %v, err = %v; want forced", forced, err)
+	}
+
+	// Well inside MinForceInterval: still missing, but no second upgrade.
+	g.now = func() time.Time { return base.Add(time.Minute) }
+	missing, forced, err := g.Check(context.Background())
+	if err != nil {
+		t.Fatalf("second Check: %v", err)
+	}
+	if forced {
+		t.Error("second Check forced inside MinForceInterval")
+	}
+	if len(missing) == 0 {
+		t.Error("second Check should still report the objects as missing")
+	}
+
+	// Past MinForceInterval: force again, because the objects are still gone.
+	g.now = func() time.Time { return base.Add(6 * time.Minute) }
+	if _, forced, err := g.Check(context.Background()); err != nil || !forced {
+		t.Fatalf("third Check: forced = %v, err = %v; want forced", forced, err)
+	}
+}
+
+// TestMissingObjectsIgnoresUnmappedKinds pins that a strategyRef whose CRD
+// is not installed is not counted as missing. No Helm re-render can create
+// such an object, so counting it would force upgrades forever.
+func TestMissingObjectsIgnoresUnmappedKinds(t *testing.T) {
+	bc := cozyDefaultBackupClass()
+	bc.Spec.Strategies = append(bc.Spec.Strategies, backupsv1alpha1.BackupClassStrategy{
+		Application: backupsv1alpha1.ApplicationSelector{Kind: "Nonexistent"},
+		StrategyRef: corev1.TypedLocalObjectReference{
+			APIGroup: apiGroup(strategyAPIGroup),
+			Kind:     "NotInstalled",
+			Name:     "cozy-default-notinstalled",
+		},
+	})
+	g, _ := newGate(t, []client.Object{sourceSecret("b"), bc},
+		helmReleaseObject(),
+		strategyObject("CNPG", "cozy-default-cnpg"),
+		strategyObject("Etcd", "cozy-default-etcd"),
+		strategyObject("Velero", "cozy-default-velero-vminstance"),
+		strategyObject("Velero", "cozy-default-velero-vmdisk"),
+		bslObject(),
+	)
+
+	missing, err := g.missingObjects(context.Background(), bc)
+	if err != nil {
+		t.Fatalf("missingObjects: %v", err)
+	}
+	if len(missing) != 0 {
+		t.Fatalf("missing = %v, want none (unmapped kind must be ignored)", missing)
+	}
+}
+
+// TestMissingObjectsSkipsBSLWhenVeleroNamespaceEmpty covers
+// velero.bslEnabled=false: the chart renders no BSL, so its absence is not
+// a defect and must not drive forced upgrades.
+func TestMissingObjectsSkipsBSLWhenVeleroNamespaceEmpty(t *testing.T) {
+	bc := cozyDefaultBackupClass()
+	g, _ := newGate(t, []client.Object{sourceSecret("b"), bc},
+		helmReleaseObject(),
+		strategyObject("CNPG", "cozy-default-cnpg"),
+		strategyObject("Etcd", "cozy-default-etcd"),
+		strategyObject("Velero", "cozy-default-velero-vminstance"),
+		strategyObject("Velero", "cozy-default-velero-vmdisk"),
+	)
+	g.VeleroNamespace = ""
+
+	missing, err := g.missingObjects(context.Background(), bc)
+	if err != nil {
+		t.Fatalf("missingObjects: %v", err)
+	}
+	if len(missing) != 0 {
+		t.Fatalf("missing = %v, want none when the BSL is disabled", missing)
+	}
+}
+
+// TestCheckSurfacesPatchFailure pins that a failed force is reported rather
+// than silently recorded as done — and that lastForce is NOT advanced, so
+// the next tick retries instead of waiting out MinForceInterval.
+func TestCheckSurfacesPatchFailure(t *testing.T) {
+	g, dyn := newGate(t,
+		[]client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()},
+		helmReleaseObject(),
+	)
+	dyn.PrependReactor("patch", "helmreleases", func(k8stesting.Action) (bool, runtime.Object, error) {
+		return true, nil, apierrors.NewForbidden(schema.GroupResource{Group: "helm.toolkit.fluxcd.io", Resource: "helmreleases"}, "backupstrategy-controller", nil)
+	})
+
+	if _, forced, err := g.Check(context.Background()); err == nil || forced {
+		t.Fatalf("forced = %v, err = %v; want an error and forced=false", forced, err)
+	}
+	if !g.lastForce.IsZero() {
+		t.Error("lastForce advanced despite the patch failing; the next tick would be throttled")
+	}
+}
+
+// TestStartDisabledWithoutHelmRelease pins the opt-out: without release
+// coordinates the runnable must return immediately instead of ticking with
+// a nil target.
+func TestStartDisabledWithoutHelmRelease(t *testing.T) {
+	g, _ := newGate(t, []client.Object{sourceSecret("b"), cozyDefaultBackupClass()}, helmReleaseObject())
+	g.HelmRelease = types.NamespacedName{}
+
+	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
+	defer cancel()
+	done := make(chan error, 1)
+	go func() { done <- g.Start(ctx) }()
+	select {
+	case err := <-done:
+		if err != nil {
+			t.Fatalf("Start: %v", err)
+		}
+	case <-ctx.Done():
+		t.Fatal("Start did not return immediately when disabled")
+	}
+}
+
+// TestCheckToleratesAbsentHelmRelease covers a plain `helm install` (local
+// development): there is no HelmRelease to force, which is not an error to
+// report every tick.
+func TestCheckToleratesAbsentHelmRelease(t *testing.T) {
+	g, _ := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()})
+
+	missing, forced, err := g.Check(context.Background())
+	if err != nil {
+		t.Fatalf("Check: %v", err)
+	}
+	if forced {
+		t.Error("forced reported true with no HelmRelease to patch")
+	}
+	if len(missing) == 0 {
+		t.Error("objects should still be reported as missing")
+	}
+}

packages/system/backupstrategy-controller/templates/_helpers.tpl

@@ -27,9 +27,22 @@
       cozystack.bucket-application + cozystack.objectstorage-controller
       ensures the controllers exist before this chart installs, but the
       BucketClaim status is reconciled asynchronously, so the first render
-      sees an unpopulated status and skips. Flux re-renders the
-      HelmRelease on its next reconcile (spec.interval); once COSI has
-      populated status.bucketName, the gated templates materialise.
+      sees an unpopulated status and skips.
+
+      The render CANNOT fail here instead: this chart is the producer of
+      the very Bucket the lookup reads, so a failed render would never
+      apply templates/bucket.yaml and the condition could never resolve.
+
+      Nor does Flux repair the skip on its own. helm-controller does not
+      re-render a release whose chart and values did not change — the
+      interval reconcile is a no-op for a healthy release, and drift
+      detection is off on operator-generated HelmReleases — so the skip is
+      PERMANENT, not a bootstrap window. Convergence is driven instead by
+      the controller's DefaultObjectsGate
+      (internal/backupcontroller/default_objects_gate.go), which forces a
+      real Helm upgrade (reconcile.fluxcd.io/forceAt + requestedAt) once
+      the bucket name is resolvable and any gated object is missing. See
+      backupStorage.reconcileDefaultObjects in values.yaml.
     - bucketNameOverride set → bypass the lookup and use it directly. This
       is the escape hatch for offline `helm template` / `--dry-run` renders
       (CI / local diffs), where lookup returns nil and no apiserver is
@@ -53,11 +66,13 @@
 {{- end -}}
 {{/* When neither path produces a value, emit the empty string.
      Strategy/BSL templates that include this helper must gate
-     themselves on a non-empty result and skip rendering until Flux
-     re-reconciles the HelmRelease (driven by spec.interval) once the
-     BucketClaim's COSI-assigned status.bucketName is populated. The
-     accompanying templates/bucket.yaml ALWAYS renders so the
-     BucketClaim CAN come into existence even on the first install. */}}
+     themselves on a non-empty result and skip rendering until a real
+     Helm upgrade re-runs this lookup with the BucketClaim's
+     COSI-assigned status.bucketName populated. That upgrade is forced by
+     the controller's DefaultObjectsGate — it does NOT happen on the
+     HelmRelease's interval reconcile. The accompanying
+     templates/bucket.yaml ALWAYS renders so the BucketClaim CAN come
+     into existence even on the first install. */}}
 {{- end -}}
 {{- end -}}
 

packages/system/backupstrategy-controller/templates/backupclass-default.yaml

@@ -6,9 +6,15 @@
   from Day 1; the Strategy CRs it references stay gated on a populated
   $bucketName because they DO need the COSI-assigned bucket name
   embedded in their template. A BackupJob fired during the bootstrap
-  window surfaces a transient Ready=False/StrategyNotReady in the
-  driver and self-heals once Flux re-renders the chart with the
-  populated BucketClaim status.
+  window surfaces a transient Ready=False/StrategyNotReady in the driver.
+
+  It does NOT self-heal on a Flux reconcile: helm-controller re-renders
+  only when the chart or values change, so the gated Strategy CRs would
+  stay absent forever. The controller's DefaultObjectsGate reads THIS
+  object's strategyRefs as the manifest of what must exist and forces the
+  real Helm upgrade that materialises them (see
+  backupStorage.reconcileDefaultObjects). Adding a route here therefore
+  also adds the referenced Strategy CR to that existence check.
 */}}
 apiVersion: backups.cozystack.io/v1alpha1
 kind: BackupClass

packages/system/backupstrategy-controller/templates/deployment.yaml

@@ -50,6 +50,25 @@ spec:
           value: {{ .Values.backupStorage.forcePathStyle | quote }}
         - name: BACKUP_STORAGE_SYSTEM_NAMESPACES
           value: {{ .Values.backupStorage.systemNamespaces | join "," | quote }}
+        {{- if .Values.backupStorage.reconcileDefaultObjects }}
+        # DefaultObjectsGate: the Strategy CRs and the Velero BSL are gated
+        # on a `lookup` of the BucketClaim this same chart creates, so a
+        # fresh install renders them empty — permanently, because
+        # helm-controller does not re-render an unchanged, successful
+        # release. The gate detects the missing objects once the bucket name
+        # is resolvable and forces exactly one real Helm upgrade on THIS
+        # release, which is the only thing that re-runs the lookups.
+        - name: BACKUP_DEFAULT_OBJECTS_BACKUPCLASS
+          value: cozy-default
+        - name: BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME
+          value: {{ .Release.Name | quote }}
+        - name: BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAMESPACE
+          value: {{ .Release.Namespace | quote }}
+        {{- if .Values.velero.bslEnabled }}
+        - name: BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE
+          value: {{ .Values.velero.namespace | quote }}
+        {{- end }}
+        {{- end }}
         ports:
         {{- if .Values.backupStrategyController.metrics.enabled }}
         - name: metrics

packages/system/backupstrategy-controller/templates/rbac.yaml

@@ -61,10 +61,15 @@ rules:
 - apiGroups: ["cdi.kubevirt.io"]
   resources: ["datavolumes"]
   verbs: ["delete"]
-# HelmReleases: pre-restore suspends HRs; post-restore rename creates new HR and deletes old
+# HelmReleases: pre-restore suspends HRs; post-restore rename creates new HR
+# and deletes old. patch additionally lets the DefaultObjectsGate stamp
+# reconcile.fluxcd.io/forceAt + requestedAt on THIS chart's own release to
+# force the Helm upgrade that materialises the lookup-gated default backup
+# objects (no list/watch: every access is a point Get through the dynamic
+# client, so no cluster-wide HelmRelease informer is started).
 - apiGroups: ["helm.toolkit.fluxcd.io"]
   resources: ["helmreleases"]
-  verbs: ["get", "create", "update", "delete"]
+  verbs: ["get", "create", "update", "patch", "delete"]
 # PVCs and PVs: pre-restore renames PVCs (delete old, create new) and patches PV reclaim policy
 - apiGroups: [""]
   resources: ["persistentvolumeclaims"]
@@ -76,6 +81,13 @@ rules:
 - apiGroups: ["velero.io"]
   resources: ["backups", "restores", "datauploads"]
   verbs: ["create", "get", "list", "watch", "update", "patch", "delete", "deletecollection"]
+# Velero BackupStorageLocation: the DefaultObjectsGate checks that the
+# cozy-default BSL — rendered by the same lookup-gated template as the
+# Strategy CRs — actually exists. Read-only, and a point Get through the
+# dynamic client, so `get` alone is enough (no informer, no list/watch).
+- apiGroups: ["velero.io"]
+  resources: ["backupstoragelocations"]
+  verbs: ["get"]
 # Velero DeleteBackupRequest: used by BackupReconciler to delete Velero backups with their storage data
 - apiGroups: ["velero.io"]
   resources: ["deletebackuprequests"]

packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml

@@ -0,0 +1,91 @@
+suite: DefaultObjectsGate wiring — the release the gate forces, and its RBAC
+
+# The Strategy CRs and the Velero BSL are rendered behind a `lookup` of the
+# BucketClaim this same chart creates, so a fresh install renders them empty
+# and helm-controller never re-renders (unchanged chart + values, drift
+# detection off) — the skip is permanent. The controller's DefaultObjectsGate
+# repairs that by forcing a real Helm upgrade on THIS release, so the release
+# coordinates it is given must be this chart's own release, and it must hold
+# the verbs to do it.
+
+templates:
+  - templates/deployment.yaml
+  - templates/rbac.yaml
+
+release:
+  name: backupstrategy-controller
+  namespace: cozy-backup-controller
+
+tests:
+  - it: "gate targets this chart's own HelmRelease and the cozy-default BackupClass"
+    template: templates/deployment.yaml
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME
+            value: backupstrategy-controller
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAMESPACE
+            value: cozy-backup-controller
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_BACKUPCLASS
+            value: cozy-default
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE
+            value: cozy-velero
+
+  - it: "velero.bslEnabled=false: the BSL is not rendered, so the gate must not look for it"
+    template: templates/deployment.yaml
+    set:
+      velero.bslEnabled: false
+    asserts:
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE
+            value: cozy-velero
+      # The rest of the gate stays wired: the Strategy CRs still need it.
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME
+            value: backupstrategy-controller
+
+  - it: "reconcileDefaultObjects=false: the gate is fully unwired (main.go disables it on an empty release name)"
+    template: templates/deployment.yaml
+    set:
+      backupStorage.reconcileDefaultObjects: false
+    asserts:
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_HELMRELEASE_NAME
+            value: backupstrategy-controller
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: BACKUP_DEFAULT_OBJECTS_BACKUPCLASS
+            value: cozy-default
+
+  - it: "ClusterRole grants patch on helmreleases (forceAt stamp) and get on backupstoragelocations"
+    template: templates/rbac.yaml
+    asserts:
+      - contains:
+          path: rules
+          content:
+            apiGroups: ["helm.toolkit.fluxcd.io"]
+            resources: ["helmreleases"]
+            verbs: ["get", "create", "update", "patch", "delete"]
+      - contains:
+          path: rules
+          content:
+            apiGroups: ["velero.io"]
+            resources: ["backupstoragelocations"]
+            verbs: ["get"]

packages/system/backupstrategy-controller/values.yaml

@@ -118,6 +118,30 @@ backupStorage:
   # rendering. Tenants are projected lazily during BackupJob reconcile.
   systemNamespaces:
     - cozy-velero
+  # reconcileDefaultObjects enables the controller's DefaultObjectsGate.
+  #
+  # The Strategy CRs and the Velero BSL below are Helm-templated behind a
+  # `lookup` of the BucketClaim THIS chart creates, because the S3 bucket
+  # name is assigned by the COSI driver (bucket-<claim-UID>) and cannot be
+  # computed at render time. On a fresh install the lookup is empty and those
+  # templates render nothing — and helm-controller does not re-render a
+  # release whose chart and values did not change (drift detection is off on
+  # operator-generated HelmReleases), so the skip is PERMANENT. Clusters have
+  # run for months with only the BackupClass present and no Strategy CRs at
+  # all; recovery needed a hand-stamped reconcile.fluxcd.io/forceAt +
+  # requestedAt, because a plain reconcile request does not re-render.
+  #
+  # With this enabled the controller resolves the bucket name from the
+  # projector's source Secret (no new dependency, no lookup), checks that
+  # every object cozy-default routes to exists, and forces one Helm upgrade
+  # when any is missing — throttled, and a no-op in the steady state. It does
+  # NOT create the objects itself: their bodies are values-driven and Helm
+  # stays their single owner.
+  #
+  # Turn this off only if you manage the default Strategy CRs out of band and
+  # accept that a missed install-time render is never repaired. Watch
+  # cozystack_backup_default_objects_missing if you do.
+  reconcileDefaultObjects: true
 # velero — BSL rendering. The cozy-default BackupStorageLocation is
 # created in cozy-velero by templates/velero-bsl.yaml so endpoint/bucket/
 # region come from the same backupStorage block used by Strategy CRs and
@@ -129,7 +153,9 @@ backupStorage:
 # Velero CRDs are guaranteed to exist by the time this chart installs.
 # The BSL template additionally gates itself on a populated BucketClaim
 # status via the bucketName helper, so on a bootstrap install the BSL
-# materialises only after the platform Bucket reconciles. Setting this
+# materialises only after the platform Bucket reconciles AND the
+# DefaultObjectsGate has forced a real Helm upgrade (an interval reconcile
+# alone does not re-render). Setting this
 # to false is an explicit opt-out for clusters that disable VM backups
 # entirely and want to avoid the cluster-default BSL — in that case the
 # cozy-default-velero-vminstance / -vmdisk strategy CRs still ship but

packages/system/bucket/templates/user-credentials.yaml

@@ -1,7 +1,40 @@
+{{/*
+  The human-friendly per-user credentials Secret is derived from the COSI
+  Secret the objectstorage sidecar writes for the matching BucketAccess
+  (packages/apps/bucket/templates/bucketclaim.yaml). That Secret carries a
+  single BucketInfo JSON document with the driver-assigned bucket name and
+  the S3 keys — none of it is knowable at render time, so a `lookup` is
+  unavoidable here.
+
+  What is NOT acceptable is silently skipping the Secret when the lookup is
+  empty. helm-controller does not re-render a release whose chart and values
+  did not change, and drift detection is off by default, so a skip at
+  install time is PERMANENT: the Secret is never created, and every consumer
+  that reads it (the dashboard, and for the platform cozy-backups bucket the
+  backupstrategy-controller credentials projector — hence every default
+  BackupClass strategy and Velero itself) stays broken until somebody forces
+  a real Helm upgrade by hand.
+
+  So fail the render instead. This release retries forever
+  (install/upgrade.remediation.retries: -1 on the <bucket>-system
+  HelmRelease, see packages/apps/bucket/templates/helmrelease.yaml) and
+  every retry re-runs the lookup, so it converges as soon as COSI publishes
+  the Secret. Failing cannot deadlock its own precondition: the BucketAccess
+  that produces the COSI Secret is rendered by the PARENT release, not by
+  this one.
+
+  requireUserCredentials: false is the escape hatch for offline renders
+  (`helm template`, `make show`, CI diffs) where lookup always returns nil
+  and no apiserver is reachable.
+*/}}
 {{- range $name, $user := .Values.users }}
 {{- $secretName := printf "%s-%s" $.Values.bucketName $name }}
 {{- $existingSecret := lookup "v1" "Secret" $.Release.Namespace $secretName }}
-{{- if $existingSecret }}
+{{- if not $existingSecret }}
+{{- if $.Values.requireUserCredentials }}
+{{- fail (printf "COSI credentials Secret %s/%s does not exist yet, so %s-credentials cannot be rendered. This is expected while the bucket is being provisioned: the release retries and converges once the objectstorage sidecar has written the Secret for BucketAccess %s. If it never appears, inspect the BucketClaim/BucketAccess in this namespace and the objectstorage-controller logs. Set requireUserCredentials=false to render this chart offline." $.Release.Namespace $secretName $secretName $secretName) }}
+{{- end }}
+{{- else }}
 {{- $bucketInfo := fromJson (b64dec (index $existingSecret.data "BucketInfo")) }}
 ---
 apiVersion: v1

packages/system/bucket/tests/user_credentials_test.yaml

@@ -0,0 +1,56 @@
+suite: per-user credentials Secret — loud failure instead of a permanent silent skip
+
+# The <bucket>-<user>-credentials Secret is derived from the COSI Secret the
+# objectstorage sidecar writes, which is only knowable through a `lookup`.
+# helm-unittest renders with no live cluster, so lookup always returns nil —
+# exactly the pre-reconcile install window these cases pin.
+#
+# The old template silently emitted nothing in that window. Because
+# helm-controller does not re-render a release whose chart and values did not
+# change, that skip was permanent: the Secret was never created, and the
+# platform credentials projector (hence every default BackupClass strategy)
+# had no source to read.
+
+templates:
+  - templates/user-credentials.yaml
+
+release:
+  name: bucket-cozy-backups
+  namespace: tenant-root
+
+tests:
+  - it: "fails the render when a declared user's COSI Secret does not exist yet"
+    set:
+      bucketName: bucket-cozy-backups
+      users:
+        system:
+          readonly: false
+    asserts:
+      - failedTemplate:
+          errorMessage: >-
+            COSI credentials Secret tenant-root/bucket-cozy-backups-system does not exist yet,
+            so bucket-cozy-backups-system-credentials cannot be rendered. This is expected while
+            the bucket is being provisioned: the release retries and converges once the
+            objectstorage sidecar has written the Secret for BucketAccess
+            bucket-cozy-backups-system. If it never appears, inspect the BucketClaim/BucketAccess
+            in this namespace and the objectstorage-controller logs. Set
+            requireUserCredentials=false to render this chart offline.
+
+  - it: "requireUserCredentials=false keeps offline renders working (no Secret, no failure)"
+    set:
+      bucketName: bucket-cozy-backups
+      requireUserCredentials: false
+      users:
+        system:
+          readonly: false
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: "no declared users: nothing to render and nothing to fail on"
+    set:
+      bucketName: bucket-cozy-backups
+      users: {}
+    asserts:
+      - hasDocuments:
+          count: 0

packages/system/bucket/values.yaml

@@ -1,2 +1,11 @@
 bucketName: "cozystack"
 users: {}
+# requireUserCredentials makes the render FAIL while a declared user's COSI
+# credentials Secret does not exist yet, instead of silently emitting no
+# <bucket>-<user>-credentials Secret. Keep it true on a live cluster: the
+# release retries (install/upgrade.remediation.retries: -1) and converges,
+# whereas a silent skip is permanent — helm-controller does not re-render an
+# unchanged release, so the Secret would never be created at all.
+# Set to false only for offline renders (`helm template`, `make show`, CI
+# diffs), where `lookup` always returns nil because no apiserver is reachable.
+requireUserCredentials: true