mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 04:33:36 +09:00
## What this PR does The cluster controller runs nine component reconcilers in sequence, and none of them logs unconditionally when it is entered. The TLS one has a line above its work, but it sits on the early return for a cluster with no security section, which `packages/apps/opensearch/templates/opensearch.yaml` never renders, and the rolling restart one opens at `V(1)`, off at the verbosity the operator ships with. So a call that stops returning leaves nothing behind at all: the operator log ends on whatever ran before it, the object stops being reconciled, and from the outside that is indistinguishable from an operator with nothing to do. That is not hypothetical. Two e2e runs on 2026-09-18 ended that way, and the evidence is the operator's own log, read out of the crust-gather host snapshot each failed run uploads, at `snapshots/opensearch/host/namespaces/cozy-opensearch-operator/v1/pod/opensearch-operator-controller-manager-*/operator-controller-manager/current.log`. In run 35343282947 the operator reconciled `OpenSearchCluster` 27 times on a steady cadence, gaps of 10.2s for almost all of them and 26.5s at the widest. Its last line is at 13:51:33.225Z. helm-controller applied the CR revision that moves the cluster onto the chart CA at 13:51:35Z. Nothing was written for the remaining four minutes, the pod stayed Running and Ready with 0 restarts, and the admin certificate the step was waiting on was never re-minted. Run 35333289958, a different branch and a different operator pod, has the same shape: last line 10:56:35.706Z, Apply at 10:56:38Z, widest gap before that 24.2s, then silence. The securityconfig Job had completed successfully in both, and the cluster itself was quiet. #4332 tracks it. A patch on the image now wraps each component so a call still in flight after a minute reports the component's own name and how long it has been running, and keeps reporting on the same interval while it is stuck. A pass through the components takes seconds on a healthy cluster and the controller requeues itself every 30s, so a minute is well past anything normal. Nothing at all is written while a component behaves, which is why this is a watchdog rather than an entry log: logging every entry would put nine lines per cluster into the log on every reconcile. The finalizer path is deliberately left alone, because every `DeleteResources` at this version returns immediately and a watchdog there could only spawn goroutines for calls that cannot block. The e2e lane already collects that log and nothing in it changes here. #4279 put a `podLogs` leg on the operator's own namespace into the opensearch catch, `namespace: cozy-opensearch-operator`, `selector: control-plane=controller-manager`, `tail: 2000`, and this branch sits on top of it. The other `podLogs` leg takes the cluster's own pods in the test namespace, and cozyreport's per-pod walk keeps only pods that are not Running or not Ready, which a stalled operator is neither, so before #4279 these lines reached the job log only through the crust-gather snapshot. This is a diagnostic, not the fix. Where the reconcile stops is still unknown, and the next occurrence should name it. The Dockerfile comment says to drop the patch once the stall itself is understood and fixed. The new code lives in `pkg/reconcilers`, already covered by the image's `go test` step. Five specs hold it: the wrapped result and error pass through untouched, a call that returns before the threshold is never reported, a call that returns before the threshold also takes the watchdog with it, the report names the component and a moving elapsed while the call is in flight, and the report repeats rather than firing once. Six mutations each fail the spec that owns them: dropping `close(done)`, `time.NewTimer(0)`, an empty component name, a frozen elapsed, a fire-once ticker, and a swallowed result. The full four-patch sequence applies to a fresh v2.8.0 tree in the alphabetical order the build relies on, and builds, vets and tests clean there, with five `-race` runs of the new specs. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: one Dockerfile comment block and one new patch file under `packages/system/opensearch-operator/images/`. No values key, schema, make target or documented workflow changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added stall monitoring for OpenSearch component reconciliation. When reconciliation remains in progress beyond the configured threshold, the operator logs the component name and elapsed time at regular intervals. - Monitoring remains silent for promptly completed operations and preserves existing results and errors. - **Documentation** - Clarified reconciliation watchdog behavior and documented why certain resource deletion operations are not monitored. <!-- end of auto-generated comment: release notes by coderabbit.ai -->