Two ends of the same path. The platform suite pins that the shared
values Secret carries networking.clusterDomain to the operator under
the subchart key, and the package suite pins that a domain set on that
key reaches the manager container as DNS_BASE, with the upstream
cluster.local default intact when nothing sets it.
Every pattern in the platform suite anchors dnsBase to its
opensearch-operator and manager parents rather than matching the key
anywhere in the payload. An unanchored pattern is satisfied by any
sibling block that happens to carry the same domain, which leaves the
assertion green after the block it is meant to pin has been deleted.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
OpenSearch refuses to start when vm.max_map_count is below 262144 and
the store may use mmap. The kernel default is 65530 and nothing in the
platform's node configuration raises it: the Talos profiles declare no
sysctls, and neither does any documented node-preparation path.
The value cannot come from the OpenSearch pod. The kernel does not
namespace vm.*, so kubelet refuses it in a pod's securityContext.sysctls
and in --allowed-unsafe-sysctls alike, both of which admit namespaced
sysctls only. The operator's own answer, spec.general.setVMMaxMapCount,
appends a privileged init container to every OpenSearch pod. Tenant
namespaces carry no PodSecurity label of their own and take the cluster
default, which Talos ships as baseline through the apiserver's
PodSecurityConfiguration; that default rejects the container, so the
StatefulSet is created and never produces a pod.
Set the value once per node instead, from a DaemonSet in the operator's
own namespace. The init container raises the limit to a floor rather
than assigning it: the tunable is node-global, so a node already tuned
higher for another mmap-heavy workload keeps its own value. The pod runs
at system-node-critical: it is a precondition for every OpenSearch pod on
the node, so it must not lose a scheduling race to the workload it
enables, though priority is not start ordering: a pod that starts first
just fails its own bootstrap check and retries. The idle container runs
the sleep under a shell with a TERM handler, because PID 1 of a
namespace receives only the signals it installed a handler for and a
bare sleep would be SIGKILLed after the full grace period on every
rollout and drain. The write
goes to /proc/sys only, with no hostPath and no edit to the node's
on-disk configuration; the pod restarts with the node and reapplies the
value. The component is declared privileged so the platform labels that
namespace enforce=privileged, which is what this DaemonSet needs and
what tenant namespaces deliberately do not get.
The file returns to a path bb660b57c emptied. That earlier revision
bind-mounted the host's /etc to append to sysctl.conf and ran an
unpinned image; this one writes no host file and pins the image by
digest.
This removes the reason for an OpenSearch pod to ask the operator for
the sysctl, but the app chart still asks. Dropping that request is a
separate change and is what stops the pods being rejected.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
#3040 set the single-replica operator Deployment to strategy.type:
Recreate to avoid two uncoordinated managers overlapping during a
rollout while leader election is gated off. That breaks Helm upgrades of
existing releases: the upgrade fails validation immediately,
helm-controller retries it (x19 over 11m in Actions run 29480724982),
and the HelmRelease never becomes Ready:
Deployment.apps "opensearch-operator-controller-manager" is invalid:
spec.strategy.rollingUpdate: Forbidden: may not be specified when
strategy type is Recreate
The trigger is server-side apply. A v1.5.x release shipped no strategy
at all, so the apiserver defaulted spec.strategy.rollingUpdate to
25%/25%, and the helm-controller field manager never owned that block.
helm-controller applies server-side by default (Install.ServerSideApply
defaults to true, Upgrade.ServerSideApply to "auto"; the operator's
package_reconciler.go sets neither), and SSA does not remove a field the
applier never owned merely because the new intent omits it -- so sending
type: Recreate leaves the defaulted rollingUpdate in place and the
apiserver rejects the merged object.
The client-side path is unaffected: DeploymentSpec.Strategy carries
patchStrategy:"retainKeys", so a 3-way merge emits
{"$retainKeys":["type"],"type":"Recreate"} and does clear rollingUpdate.
That is why this surfaces through helm-controller's server-side apply
but not a local 'helm upgrade'.
Keep type: RollingUpdate and set rollingUpdate.maxSurge: 0 /
maxUnavailable: 1 on the single-replica path instead. maxSurge: 0 caps
total pods at replicas, so the old manager is scaled down before the new
one is scaled up, rather than starting a surge pod while the old manager
is still fully live as the default 25% would. It is not Recreate's
exclusivity -- the deployment controller orders the ReplicaSet desired
counts, not the pod lifecycles, so a draining old manager can still
overlap the new pod -- but it keeps the type transition legal on every
apply path, and recovers releases already on an rc build with Recreate.
This matches the existing kube-ovn idiom (central-deploy.yaml).
Also add a 'test' target to the package Makefile:
hack/helm-unit-tests.sh runs a package's suite only when its Makefile
defines one, so tests/leader_election_test.yaml had never executed.
Multi-replica installs keep the default RollingUpdate (leader election
on), unchanged. The change is carried in patches/leaderElection.diff so
it survives 'make update'.
Verified on kind v1.33.1: SSA of the v1.5.x manifest followed by SSA of
the Recreate manifest reproduces the error, while the same pair applied
client-side succeeds; SSA of the maxSurge:0 manifest succeeds both from
v1.5.x and from a live Recreate Deployment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The operator Deployment ships a single replica but always passes
--leader-elect to the manager. On one replica leader election adds no HA
value: a transient apiserver/etcd lease-renewal blip makes
controller-runtime self-terminate (os.Exit on "leader election lost"),
which crashloops the manager and surfaces at install time as a
Deployment stuck in Failed.
Gate --leader-elect on the replica count via a manager.replicaCount
value (default 1). The single-replica install now rides out
control-plane latency instead of crashlooping, and scaling to more than
one replica re-enables leader election automatically for genuine HA.
Also set the Deployment strategy to Recreate on a single replica: with
leader election off, a default RollingUpdate would briefly run two
managers (maxSurge) during an operator upgrade, both reconciling without
lease coordination. Recreate closes that window at zero availability
cost, while installs scaled past one replica keep RollingUpdate.
The change to the vendored opensearch-operator 2.8.0 chart is carried in
patches/leaderElection.diff and re-applied by 'make update' so it
survives a chart refresh.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>