Ivan Okhotnikov 7b08fcdca9 fix(operator): unify recovery gating through state machine, close source-watcher-unresponsive dormancy loop
Self-review on 467b788b8 caught a MAJOR bug introduced by the separate
awaitSourceWatcherResponse helper: if source-watcher never responds to
a forced drift (network partition, missing predicate, dead pod), our
own reasonRecoveryForced marker keeps reflecting back via Owns() forever,
and the wait-forever helper schedules a 1-second requeue tight loop
without ever advancing the retry state machine. SourceWatcherStalled
give-up was unreachable.

Fix: remove awaitSourceWatcherResponse and unify all three recovery
entry conditions — our own marker, source-watcher's Progressing during
rebuild, and the classic first-time stuck detection — through a single
gate that routes to maybeRecoverArtifactGenerator. The state machine's
decideRecovery is the single arbiter of wait/force/give-up, and its
Wait branch is bounded by the same backoff schedule as Force. When
backoff elapses on a still-reflecting own-marker the machine escalates
to another force (bumps attempts, re-stamps requestedAt); when attempts
exhaust it surfaces SourceWatcherStalled honestly.

The own-marker filter also no longer requires attempts>0, so an
operator crash between step 1 (status patch) and step 2 (metadata
patch) of forceArtifactGeneratorDrift is still routed through the
state machine on the next reconcile rather than leaking the synthetic
marker onto the PackageSource.

Also address the non-blocking notes from the same self-review:

* Docstring on maybeRecoverArtifactGenerator was still describing the
  removed reset-counter branch and the "requestedAt alone is a no-op"
  framing. Rewrite to explain the current shape: status write triggers
  the drift signal, requestedAt bump triggers the reconcile event —
  both are required, neither alone works.
* TestForceArtifactGeneratorDrift_MetadataPatchFailure_RollbackAnnotations
  now asserts reconcile.fluxcd.io/requestedAt is also rolled back on
  the metadata-patch failure branch (symmetric with the tracking
  annotations).
* TestUpdateStatus_OwnMarkerRoutesToAwait asserts an exact 25s
  RequeueAfter (backoffFor(1)=30s minus 5s elapsed) so a silent
  regression in the backoff arithmetic is caught.

New regression tests for the source-watcher-unresponsive path:

* TestUpdateStatus_OwnMarkerStale_TriggersRetry: with attempts=1 and
  lastRecoveryAt older than backoffFor(1), the reconcile must
  escalate through the Force branch and bump attempts to 2 rather
  than idle in a tight requeue loop.
* TestUpdateStatus_OwnMarkerAttemptsExhausted_GivesUp: with
  attempts=maxRecoveryAttempts and the marker still reflecting back,
  the reconcile must set PackageSource Ready=False /
  SourceWatcherStalled instead of extending the loop indefinitely.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-05 08:30:19 +03:00
2025-10-08 09:43:34 +05:00
2025-04-01 18:48:14 +02:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

Open Source Apache-2.0 License Support Active GitHub Release GitHub Commit OpenSSF Best Practices

Cozystack

Cozystack is a free platform and framework for building clouds.

Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 47.2%
Shell 37.6%
TypeScript 7%
Go Template 5.4%
Makefile 1.4%
Other 1.2%