mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
Aleksei Sviridkin
e89617d754
## What this PR does The opensearch-operator resolves every in-cluster name it dials as `<svc>.<ns>.svc.<DNS_BASE>`, and nothing set `DNS_BASE`, so it stayed on the vendored chart default `cluster.local` while the platform runs on a configurable domain. On a cluster that is not on `cluster.local` the securityconfig bootstrap job waits forever on a name that never resolves, the security configuration is never applied, and OpenSearch nodes stay unready. That is issue #4073. The platform now carries the domain to the operator in the `cozystack-values` Secret, under the subchart key the vendored chart already reads: ```yaml opensearch-operator: manager: dnsBase: "cozy.local" ``` `charts/opensearch-operator` reads that as `.Values.manager.dnsBase` and hands it to the manager as `DNS_BASE`, so nothing under `charts/` changes and the package needs no patch. The value renders from `networking.clusterDomain` rather than a literal, so a cluster that keeps `cluster.local` keeps working, and a package rendered without the Secret keeps the upstream `cluster.local` default. This is the route the KEDA cluster domain already travels through the same Secret. The whole change is one key in `packages/core/platform/templates/apps.yaml` plus two test suites: one pinning that `networking.clusterDomain` reaches the Secret under that key, one pinning that the key reaches the Deployment env in the package. ### Why this is not the `global` route Earlier revisions of this branch, and #3357 before it, emitted the domain as `global.clusterDomain` and patched the vendored template to read it. `global` is the one key Helm copies into every subchart, which is why it looked right. That is also the problem. The `cozystack-values` Secret is `valuesFrom` on every HelmRelease the operator builds unless the PackageSource opts out with `operator.cozystack.io/skip-cozystack-values` (`internal/operator/package_reconciler.go:265-273`), so a new key under `global` lands in the values of nearly every release on the cluster. One of them broke on it. The vendored smee chart in `packages/system/bootbox` gates on the truthiness of the whole `global` map and then indexes `global.rbac`, so a single unrelated key under `global` ends its render. This still reproduces on `main`: ```console $ helm template bootbox packages/system/bootbox # exit 0 $ helm template bootbox packages/system/bootbox --set global.clusterDomain=cozy.local Error: cozy-bootbox/charts/smee/templates/role.yaml:4:32 executing "cozy-bootbox/charts/smee/templates/role.yaml" at <.Values.global.rbac.type>: nil pointer evaluating interface {}.type ``` The subchart key reaches the operator with no patch, no change to any other release, and nothing to harden in bootbox. I measured that rather than reasoning about it: rendering all 170 charts under `packages/*/*` plain and with `--set opensearch-operator.manager.dnsBase=cozy.local`, exactly one chart's output changes, opensearch-operator, by the one `DNS_BASE` line this PR is about. Nine other charts differ between the two runs by exactly as much as two identical runs of the same command differ, which is the random material Helm regenerates per render, and thirty-five fail in both modes for reasons unrelated to the flag, with byte-identical errors either way. The smee hardening is still worth having, because the next key anyone puts under `global` hits the same gate, and that gate is still on the default branch of tinkerbell/charts. It is not carried here. It is available as a separate change against its own package. ### On having more than one way to say the cluster domain Review feedback on the earlier revision asked for the opposite of this: fold the `keda:` block into `global.clusterDomain` so the platform has one answer rather than two added a fortnight apart for the same reason. The instinct is right and I am not doing it here, because `global` is the channel that carries the cost. A key under it lands in the values of nearly every release, one vendored chart already ends its render on it, and the next chart with the same gate breaks on the day someone adds a second key. The subchart key has no reach: the chart that declares it reads it and everything else ignores it, which is how the KEDA domain already travels. That does leave the repo with several ways to tell a chart its domain, and that complaint stands. What it avoids is a shared channel whose blast radius grows with every new user of it. Whether the platform should grow one general mechanism here, and which one, is worth settling on its own rather than inside a bug fix. ### Why the stuck Job is a manual step and not a migration The release note asks operators to delete a wedged `<release>-securityconfig-update` Job once. On an existing OpenSearch installation that Job is already stuck, so this is a step for everyone the fix is for, and `packages/core/platform/images/migrations/` is the mechanism built for exactly that. It is not in this PR because the migration deletes Jobs in tenant namespaces, which is a wider grant and a wider blast radius than the one key this change adds, and the fix works without it. If it should ride the same release, it is a separate commit and I will add it. ### Upgrade behaviour on a running cluster Read from the operator source at v2.8.0. `DNS_BASE` is read on every call through `helpers.ClusterDnsBase` (`pkg/helpers/constants.go:35-41`), so the manager restart is the whole switch. On a cluster that already runs OpenSearch clusters: - Certificates are not re-issued. Transport and http certs are generated only when their Secret is missing (`pkg/reconcilers/tls.go:272-295`, `490-518`), per-node transport certs only for a pod with no entry in the Secret (`tls.go:373-377`), the admin cert only when missing, expiring within five days or not signed by the CA (`tls.go:176-211`). Existing SANs keep the old suffix. Nothing in-cluster verifies them: the securityadmin job runs `curl -k` and `securityadmin.sh -nhnv` (`pkg/reconcilers/securityconfig.go:34,40,46`), the operator client sets `InsecureSkipVerify` (`opensearch-gateway/services/os_client.go:77`), Dashboards run with `opensearch.ssl.verificationMode: none` (`pkg/builders/dashboards.go:180`). - OpenSearch node pods do not roll, because no StatefulSet field derives from the DNS base. Dashboards roll, since `OPENSEARCH_HOSTS` changes (`pkg/builders/dashboards.go:46-47`). - A securityconfig Job already stuck in its wait loop is not recreated. The reconciler keeps an existing `<release>-securityconfig-update` Job while its checksum annotation matches the securityconfig Secret (`securityconfig.go:140-146`), and the target hostname is not part of that checksum. Delete that Job once by hand. The next reconcile recreates it against the new name, and users and roles start applying from there. ### Known and not addressed here The upstream `manager.dnsBase` knob stays usable. The platform writes it through `spec.valuesFrom`, and a per-package override in `Package.spec.components[].values` lands in `spec.values`, which Flux merges on top of the references, so a package that needs a different domain can still say so and win. The fallback differs from its neighbours and that is deliberate. `packages/apps/opensearch` falls back to `cozy.local`, `packages/system/keda` declares `cozy.local` as its package default, and this package stays on the upstream `cluster.local`. Inside the bundle all three receive the same value, so nothing diverges in practice, and `cluster.local` is the right standalone default for a Cozystack installed over an existing kubeadm or k3s cluster. A cluster domain that YAML reads as a number does not survive the round trip. The platform quotes the value into the Secret, so `networking.clusterDomain: 123` arrives as the string `123`, but the upstream template renders it as a bare `value: 123` in the container env, and an `EnvVar` value has to be a string. Quoting it means patching the vendored template, which is the thing this approach removes, and an all-numeric cluster domain is not a usable DNS name to begin with. Before this change the key was never set from the platform, so the path is new here even though nothing reaches it. The vendored `opensearch-operator` Deployment template does not reproduce from `make update` and has not for a while: a fresh pull plus the patch differs in three places, and one of them is a live defect. The committed `nindent 10` on `manager.extraEnv` renders invalid YAML, so that knob cannot be used at all, while upstream's `nindent 8` works. Nothing in the tree sets `extraEnv`, so it is latent, and the drift predates this branch. This PR does not touch that patch at all and does not carry a fix for it. Nothing in CI renders a package chart with the values the platform injects, which is why the smee breakage on the earlier approach had to be found by hand. That is a gap in the harness. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff, which is one template key in `packages/core/platform/templates/apps.yaml` and two test files. `packages/core/platform/values.yaml` is untouched and gains no key, so the hand-written platform-package table on the website is not affected. No package, app schema, version enum, CRD, release prefix or naming convention moves, so the Terraform provider is not reached. The `networking.clusterDomain` key already exists and its default does not change, so ansible-cozystack, which passes `--cluster-domain=cozy.local` to k3s and sets only the networking CIDRs in its platform-package template, needs nothing: the operator now follows the domain that repo already configures. talm sets the Talos `cluster.network.dnsDomain` from its own value and does not write the platform value either. Nothing under `hack/`, no namespace and no package-tree layout moves, so ccp is not reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): the operator's DNS_BASE now follows the platform cluster domain (`networking.clusterDomain`) instead of the upstream `cluster.local` default, so the securityconfig job and OpenSearch Dashboards reach the cluster on installations that use `cozy.local`. On upgrade, an existing `<release>-securityconfig-update` Job that is already stuck waiting on the old hostname is not recreated by the operator: delete it once by hand and the next reconcile recreates it against the new hostname. Existing certificates keep their SANs and keep working, nothing in-cluster verifies them. ```
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.