mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-23 03:27:50 +09:00
fix(cluster-api): resync capi-operator every minute (#4939)
## What this PR does After an upgrade restarts capi-operator, a Cluster API provider can keep `Ready=False` for about 20 minutes while its controller is already running, and `capi-providers-core` times out waiting for the CoreProvider. The bundled operator v0.19.0 drops create events of provider Deployments it finds running after a restart (kubernetes-sigs/cluster-api-operator#1031), so it looks at them again only when one changes or on informer resync, and the first resync after a restart comes two sync periods later, about 20 minutes with the default 10m. This sets the operator `--sync-period` to 1m. On a test cluster a provider left `Ready=False` across an operator restart recovered 2m02s after the operator took its lease, the operator with the default period had not recovered after more than 4 minutes. A resync is served from the operator cache and writes nothing while providers are in sync: no provider object was updated over five resyncs, about 15 log lines a minute, 2m CPU and 26Mi memory. The vendored chart has no value for this flag, so a small package patch (reapplied by `make update`) adds one, and a test holds the rendered argument. The upstream fix is in cluster-api-operator v0.27.0, bumping to it is a bigger change and is left for a later release. Mitigates #4926. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(cluster-api): resync capi-operator every minute so a provider left not ready across an operator restart recovers in about two minutes instead of about twenty ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * The Cluster API operator now refreshes its cached state every minute. After an operator restart, provider readiness status can recover sooner instead of remaining stale for a longer period. * Providers that are already in sync are not written back to the API during these cache refreshes. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
5 files changed
+51
No files matched your search
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bats
|
||||
# The operator's resync period is what bounds the stale-Ready window after an
|
||||
# operator restart (kubernetes-sigs/cluster-api-operator#1031): about two
|
||||
# periods. It reaches the manager only through patches/sync-period.diff, which
|
||||
# `make update` has to reapply on every re-vendor, and through the value in
|
||||
# values.yaml. Losing either silently returns the window to about 20 minutes,
|
||||
# so this checks the argument the rendered Deployment actually carries.
|
||||
|
||||
load test_helper
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||
OPERATOR="$REPO_ROOT/packages/system/capi-operator"
|
||||
|
||||
@test "the operator manager runs with a one-minute resync period" {
|
||||
local rendered args
|
||||
rendered="$BATS_TEST_TMPDIR/operator.yaml"
|
||||
helm template capi-operator "$OPERATOR" --namespace cozy-cluster-api > "$rendered"
|
||||
args="$(yq -N 'select(.kind == "Deployment") | .spec.template.spec.containers[] | select(.name == "manager") | .args[]' "$rendered")"
|
||||
[ -n "$args" ] || { echo "no manager container args rendered from $OPERATOR" >&2; return 1; }
|
||||
case "$(printf '%s\n' "$args" | grep -c '^--sync-period=')" in
|
||||
1) ;;
|
||||
*) echo "expected exactly one --sync-period in the manager args, got:" >&2; echo "$args" >&2; return 1 ;;
|
||||
esac
|
||||
printf '%s\n' "$args" | grep -qx -- '--sync-period=1m' || { echo "manager args: $args" >&2; return 1; }
|
||||
}
|
||||
@@ -9,3 +9,6 @@ include ../../../hack/package.mk
|
||||
|
||||
update: clean capi-operator-update
|
||||
rm -rf charts/cluster-api-operator/charts/
|
||||
# The upstream chart has no value for --sync-period; values.yaml explains
|
||||
# why we set it.
|
||||
patch -p1 < patches/sync-period.diff
|
||||
@@ -86,6 +86,9 @@ spec:
|
||||
- --leader-elect-retry-period={{ .retryPeriod }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- with .Values.syncPeriod }}
|
||||
- --sync-period={{ . }}
|
||||
{{- end }}
|
||||
command:
|
||||
- /manager
|
||||
{{- with .Values.image.manager }}
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
--- a/charts/cluster-api-operator/templates/deployment.yaml
|
||||
+++ b/charts/cluster-api-operator/templates/deployment.yaml
|
||||
@@ -86,6 +86,9 @@
|
||||
- --leader-elect-retry-period={{ .retryPeriod }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
+ {{- with .Values.syncPeriod }}
|
||||
+ - --sync-period={{ . }}
|
||||
+ {{- end }}
|
||||
command:
|
||||
- /manager
|
||||
{{- with .Values.image.manager }}
|
||||
@@ -1,4 +1,12 @@
|
||||
cluster-api-operator:
|
||||
# A restarted v0.19 operator ignores the provider Deployments it finds
|
||||
# running (kubernetes-sigs/cluster-api-operator#1031), so a stale
|
||||
# Ready=False on a provider clears only at the first informer resync,
|
||||
# two periods after the restart: ~20 min at the 10m default, ~2 min at
|
||||
# 1m (both measured). A resync is served from the cache and changes
|
||||
# nothing when providers are in sync, so the short period costs no API
|
||||
# writes. Applied through patches/sync-period.diff.
|
||||
syncPeriod: 1m
|
||||
resources:
|
||||
manager:
|
||||
limits:
|
||||
|
||||
Reference in new issue
Block a user