myasnikovdaniil
6b6fad03fc
## Summary `cozystack-platform` HelmRelease times out at 15m on fresh install when a PackageSource's derived ArtifactGenerator stalls in `Ready: Unknown` even though its ExternalArtifacts are produced and consumed successfully. Root cause: fluxcd source-watcher patches `.status` and `.status.conditions` as separate apiserver requests via `patch.Helper`, so an etcd timeout under first-install load can leave `Inventory` + `ObservedSourcesDigest` persisted while the Ready-condition write is lost. Upstream tracker: [fluxcd/pkg#934 — `patch.Helper` creates race conditions in downstream controllers](https://github.com/fluxcd/pkg/issues/934). The bug is present in the currently-deployed v2.1.0 (pinned at `internal/fluxinstall/manifests/fluxcd.yaml:8230`), and a source-watcher version bump alone does not fix it because the race lives in the shared `fluxcd/pkg` runtime. ## What this PR does `PackageSourceReconciler` now detects the stuck ArtifactGenerator signature (Inventory populated, ObservedSourcesDigest set, Ready=Unknown on the current generation, LastTransitionTime older than a 30s grace period) and forces source-watcher onto its drifted-branch reconcile by writing **both** signals it keys on: 1. **`Ready=False` on the AG's status subresource.** source-watcher's `detectDrift` (v2.1.0 `artifactgenerator_drift.go:52`) treats `IsFalse(Ready)` as `NotReady` drift, promoting the reconcile past the no-drift early-return at `artifactgenerator_controller.go:164`. 2. **`reconcile.fluxcd.io/requestedAt` bump on the AG's metadata.** source-watcher's ArtifactGenerator watch is registered with `predicate.Or(GenerationChangedPredicate, ReconcileRequestedPredicate)` — the annotation is what enqueues the reconcile in the first place. Without it a status-only change never fires the watch and the AG would sit until the periodic self-requeue (hardcoded to 1h in `GetRequeueAfter`), well outside the 15m HR install timeout. Both signals are load-bearing — either alone is a no-op. The status write triggers the drifted branch inside source-watcher's reconcile; the annotation write triggers the reconcile that inspects that drift. On successful rebuild source-watcher writes `Ready=True` at `controller.go:254`; on a genuine rebuild failure it writes `Ready=False` with a real reason. Either way an honest condition from upstream replaces our synthetic False, and the `Owns(&ArtifactGenerator{})` watch fires this reconciler to copy the resolved condition onto the PackageSource. Bounded-retry harness sizes the total budget at ~11.5m (`30s + 60s + 2m + 4m + 4m`) safely inside the 15m HR install timeout; after `maxRecoveryAttempts` fruitless force-drift attempts the driver surfaces `PackageSource.Ready=False, reason=SourceWatcherStalled` so an operator can intervene, rather than lying about the state. Retry state (attempt count + last-force timestamp) lives on the AG as `cozystack.io/source-watcher-recovery-*` annotations so it survives operator restarts. Counter reset is via the canonical `clearRecoveryTracking` on the not-stuck path (Ready=True from source-watcher, or Ready=False with an upstream reason), NOT keyed on `LastTransitionTime` — source-watcher's Progressing write during rebuild always advances that timestamp, and a heuristic reset there would make `SourceWatcherStalled` unreachable under the very race this driver targets. ## Owns()-re-entrancy handling `Owns(&ArtifactGenerator{})` re-fires this reconciler on the force-drift's own writes (both the status patch and the metadata patch). A unified gate in `updateStatus` routes three overlapping signals into the same state machine: - **Our own marker reflecting back** — `Ready.Reason` carries `SourceWatcherRecoveryForced` (written only by `forceArtifactGeneratorDrift`). - **source-watcher's Progressing during rebuild** — Ready=Unknown on the current generation, tracking annotations already present. - **Classic first-time stuck detection** — the fluxcd/pkg#934 signature past the grace period. All three route to `maybeRecoverArtifactGenerator`, whose `decideRecovery` state machine is the single arbiter of wait vs re-force vs give-up. The alternative — a separate "wait for source-watcher" helper — was tried and rejected: if source-watcher never responds (missing predicate, network partition, dead pod), a wait-only path tight-loops without ever advancing to SourceWatcherStalled. The unified gate lets the state machine's own backoff schedule bound both the wait and the eventual give-up. ## Contract `PackageSource.status.conditions[Ready]` contract is **unchanged**: - `True` still means "source-watcher processed the current revision" (copied through from AG on the fast path). - `False` from a genuine upstream failure still passes through unchanged. - `False, reason=SourceWatcherStalled` is new — surfaces after `maxRecoveryAttempts` fruitless force-drift attempts, an honest operator-actionable signal rather than a false-positive True. - During force-drift and backoff windows, `Ready=Unknown, reason=AwaitingSourceWatcherRecovery` describes what the operator is waiting on. - The synthetic `SourceWatcherRecoveryForced` marker is an operator-internal signal on the AG only; it never leaks to the PackageSource. ## Retirement Once `fluxcd/pkg#934` lands and source-watcher consumes a `patch.Helper` that either serialises or transactionally combines the `.status` / `.status.conditions` writes, this whole recovery driver can be deleted and `updateStatus` can copy the AG's Ready condition through unconditionally. A `TODO(remove once fluxcd/pkg#934 lands and is rolled out)` marker on the driver points at the upstream issue. ## HA note The recovery driver stores its state as annotations on the ArtifactGenerator, so it races on write when two cozystack-operator replicas both see the same stuck target. Deploy the operator with `--leader-elect=true` (or `replicas: 1`, which is the shipped default in `packages/core/installer/templates/cozystack-operator.yaml`). ## Testing **Unit tests** cover: stuck-detection predicate with grace-period cases, bounded-retry state machine (Force / Wait / GiveUp branches), backoff schedule, tracking-annotation parser hardening, mutation contract on the two-subresource force-drift shape (success, status-patch failure rollback, metadata-patch failure with Ready=False persisted and annotations rolled back, rollback-restores-prior-bump), no-op clear-tracking, Reconcile early-return on DeletionTimestamp, follow-up scheduling on Unknown-within-grace to avoid the 10h informer resync dormancy, Owns()-re-entrancy for both our own marker and Progressing during recovery, canonical reset on Ready=True, and the source-watcher-unresponsive path (stale own-marker triggers retry through Force, exhaustion surfaces as SourceWatcherStalled). All pass with `-race`; `go vet` and `golangci-lint` clean. **E2E on dev3** — pending on the current tip. An earlier e2e ran on `97b054bbb`, which observed a stuck AG recovering after our force-drift but also had a dormancy bug (fixed since) and the requestedAt bump missing (fixed since). Both of the current shape's load-bearing mechanisms — requestedAt trigger + status-condition drift — need verification against a live source-watcher on a fresh e2e run before merge.
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.