A Velero RestoreJob is about to require its Velero Backup to exist and
be Completed before it halts the application. Without one in the
fixture, the first pass of this test fails the RestoreJob before it
suspends the HelmRelease, so the test would stop exercising the stale
cache it is about.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
The BackupJob and RestoreJob reconcilers decide a job is still active
from the informer cache, and the cache can trail their own writes. A
driver's first pass stamps StartedAt, which requeues the job, and may
then fail the job in the same pass on a transient error, such as a 503
from the aggregated apps.cozystack.io API while it reads the
application. The requeued pass can start before the informer has seen
Failed: it retries the step that failed and, once the error has
cleared, creates the driver object anyway. Its own status write then
conflicts, the pass after it sees Failed and skips the job, and a
Velero Backup is left running that no Backup artifact will reference
and nothing tracks.
Against a real apiserver and the manager's cache, with one 503 on the
first application read of each job, 5 to 10 BackupJobs out of 40 ended
Failed with a Velero Backup.
A restore pass that starts on such a stale copy does worse, since the
step it retries can be destructive: suspending the VM's HelmRelease and
halting the VM before creating a Velero Restore, or patching the
Postgres application and purging its cluster.
Both reconcilers now re-read the job through the manager's uncached
reader when the cached copy says it is still active, and skip it when
the apiserver holds it as finished. The check sits ahead of the
dispatch, so it covers every driver, and a job the cache already
reports as finished costs no extra request.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Mattia Eleuteri