mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
Aleksei Sviridkin
325a4e3cee
## What this PR does A single-replica OpenSearch release stops rolling once any index on it asks for a replica. The copy has nowhere to go on one data node, so the cluster stays yellow for good. The operator's restart gate at v2.8.0 opens on green, or on yellow only when `.opensearch-observability` is the yellow index, and at OpenSearch 2.11.1 that index is `auto_expand_replicas: 0-2`, green on one node. So the gate never opens. Every release gets such an index by itself. The operator sets `plugins.security.audit.type: internal_opensearch`, and the first audited event creates a daily `security-auditlog-*` index with one replica. After that nothing that needs a restart is rolled out, a version upgrade or the move of the HTTP listener onto the cert-manager chain included. This is what timed out the `opensearch` e2e suite on some runs. Both failed runs I read show the same thing in the OpenSearch log: the bootstrap node leaves, `Connection reset` on the transport layer, and `[security-auditlog-<date>] creating index ... shards [1]/[1]` within 40 ms. The security plugin audits that exception (`SecuritySSLNettyTransport` hands it to `AuditLogSslExceptionHandler`), so the reset is what creates the index. Then the operator logs `waiting for health to be green` every ten seconds until the assert gives up. Not sure why the green runs had no audit index before the roll, a green run keeps no snapshot. The fix carries opensearch-project/opensearch-k8s-operator#1569 back to v2.8.0 as a second patch on the in-tree operator image, with the upstream tests. The new gate lets a yellow cluster through once allocation is enabled, every data node has joined and no shard is moving, and `PreparePodForDelete` refuses to restart a node that holds the only active copy of a shard meant to have others. A lone data node is exempt. I left out the third commit of that PR (tiered data roles in `HasDataRole`), the chart renders only the `data` role. A bump would not help: the only operator build after v2.8.0 is `3.0.0-alpha` from January, and its gate holds this case the same way. The patch goes away on the first release that includes #1569. The suite now creates an index with an unassignable replica before it moves the listener, so the roll always runs against a yellow cluster. The catch block also reads cluster health and the shard table from the node, because the operator logs that health held a restart back and never what the health was. The image digest in `values.yaml` stays as it is. Closes #4259 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: an e2e suite, the operator image's Dockerfile header and patch directory, one comment in the operator package's `values.yaml`. No values key, schema, make target or documented workaround changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): a single-replica OpenSearch release no longer gets stuck ahead of every rolling restart once an index with a replica, such as the security audit log or any index created with default settings, leaves the cluster permanently yellow ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved rolling restart handling for clusters that remain yellow, allowing restarts to proceed safely once shard activity has settled. - Prevented restart operations from becoming blocked by temporary shard-allocation settings. - Added safeguards to ensure data is not disrupted while replicas are relocating or initializing. - **Diagnostics** - Added automatic cluster health and shard diagnostics when restart-related test operations time out or encounter issues. <!-- end of auto-generated comment: release notes by coderabbit.ai -->