Files
IvanHunters b222cb6580 fix(opensearch-operator): keep a single-replica cluster electable after bootstrap
The operator deletes the bootstrap pod as soon as initialization completes and
never looks at the voting configuration first. With one cluster_manager replica
there are exactly two eligible nodes, OpenSearch keeps the configuration at an
odd size of one, and the voter it keeps is the bootstrap node that formed the
cluster. Deleting that pod takes away the only vote there was, and the survivor
can never hold an election again, because changing the configuration itself
needs a quorum. The cluster is unusable about a minute after it installs, and a
node with no cluster-manager still answers the readiness probe, so nothing says
so until something restarts the pod.

Build the operator in-tree, the way kamaji and redis-operator already are, and
carry a patch that excludes the bootstrap node from the voting configuration and
waits for that to commit before deleting the pod. Reaching the cluster is a
precondition rather than a nicety: every failure keeps the pod and requeues,
because a bootstrap pod that outlives its purpose costs a pod, while a cluster
that cannot elect does not recover without a human.

Deleting a pod is a request rather than a wait, so clearing the exclusion
afterwards can find the node still in the cluster and fail. That path requeues
and is retried once the pod is really gone, which keeps a stale entry out of the
capped exclusion list; clearing stays at the API default wait_for_removal=true
so it cannot re-admit a node some other operation is still draining.

Upstream carries the same defect, tracked as
opensearch-project/opensearch-k8s-operator#1448.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-09-14 18:45:41 +03:00

33 lines
1.4 KiB
Makefile

# Single source for the version: the image's ARG VERSION. The chart and the
# binary have to move together, because the chart ships the CRDs the binary is
# compiled against.
OPENSEARCH_OPERATOR_TAG=$(shell grep -F 'ARG VERSION=' images/opensearch-operator/Dockerfile | cut -f2 -d=)
export NAME=opensearch-operator
export NAMESPACE=cozy-$(NAME)
include ../../../hack/common-envs.mk
include ../../../hack/package.mk
update:
rm -rf charts
helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/
helm repo update opensearch-operator
helm pull opensearch-operator/opensearch-operator --version $(patsubst v%,%,$(OPENSEARCH_OPERATOR_TAG)) --untar --untardir charts
patch --no-backup-if-mismatch -p1 < patches/leaderElection.diff
image:
docker buildx build images/opensearch-operator \
$(call image-tags,opensearch-operator,$(OPENSEARCH_OPERATOR_TAG)) \
$(call cache-args,opensearch-operator) \
--metadata-file images/opensearch-operator.json \
$(BUILDX_ARGS)
REPOSITORY="$(REGISTRY)/opensearch-operator" \
yq -i '.opensearch-operator.manager.image.repository = strenv(REPOSITORY)' values.yaml
TAG=$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/opensearch-operator.json -o json -r) \
yq -i '.opensearch-operator.manager.image.tag = strenv(TAG)' values.yaml
rm -f images/opensearch-operator.json
test:
helm unittest .