mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-12 23:54:01 +09:00
f48a94de6d53d23801f4a27b016090e03ff24632
582
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ZverGuy
|
bbdf05c6c7 |
feat(platform): deploy the observability-proxy
The proxy only answers for stores that the monitoring stack runs, so it follows monitoring-agents: on by default in the full variants and opt-in through enabledPackages on the slim ones, where monitoring is opt-in too. It is a stateless reader with no CRDs, so networking is its only dependency. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
myasnikovdaniil
|
4087218a02 |
docs: move the repo-only guides onto the website (#5034)
## What this PR does Seven guides under `docs/` were operator facing and written alongside the code, and left in the source tree where no user looks. They are on the website now, so this deletes them here and repoints every reference. ``` docs/operations/backup-classes.md -> operations/services/backup-classes docs/operations/resource-presets.md -> guides/resource-management docs/storage-immutability.md -> operations/storage-class-immutability docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits docs/gateway-classes.md -> networking/gateway-classes docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups ``` `docs/release.md`, `docs/deprecating-applications.md` and the two SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`, a migration and a bats suite read them by path, which is the test from #3777. References point at `https://cozystack.io/docs/v1/...`. The site rewrites that short prefix to the latest version in `layouts/404.html`, so the links follow releases and pin none. Backup Classes, resource management and `storageClass` immutability exist in the latest version already. The other four pages are only in `next/` until 1.7.0 is promoted, and their links start to resolve then. The generated mongodb files come from `values.yaml` through `make generate` with cozyvalues-gen v1.7.0, the version pinned in `.github/workflows/pre-commit.yml`. Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114) and kubernetes-nodes (223), go tests for backupcontroller, presets and the operator, rd-presets-check, the migration-50 bats suite. ### Downstream repositories - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#763 Walked the rest of the trigger map against the diff. No provider follow-up: the mongodb `values.schema.json` diff is one `description` string, with no field added, removed or renamed, no enum, no default, and no `kind`, `plural` or `release.prefix` change. `packages/core/platform/values.yaml` changed in comments only and no key, so the hand written platform-package table on the site is unaffected. Please merge after cozystack/website#763, the links in the code only resolve once those pages deploy. ### Release note ```release-note docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there. ``` |
||
|
Myasnikov Daniil
|
59932e5098 |
fix(platform): report FoundationDB patch drift in migration dry runs
The carry path skipped diagnostics in dry-run mode, so its summary could report no releases needing attention even when the upgrade would lower a FoundationDB patch version. Skip only the write and share the warning and summary bookkeeping with a real run. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
4cb505a512 |
fix(platform): record 7.1 FoundationDB clusters instead of carrying them
The operator no longer carries the 7.1 client, and the chart no longer offers v7.1, so a 7.1 cluster has no line to be carried to. Migration 59 now records it, together with any release whose version already names a line that is no longer offered; both have to be upgraded or repointed before the chart will render them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
d5ebb62632 |
fix(platform): keep migration 60's record whole across a retry
A failed patch stops the migration before the stamp, and the retry finds version already written on the releases it carried, so it never reached the step that records a patch-level move. The move is now worked out from the values alone, so a retried run records it too. An OpenSearch release storing version: null rendered the old chart's v2 default and is refused by the new one; it now goes on the record before it turns up as a stuck release. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
8c2e20539d |
fix(platform): record every release migration 60 moves
A FoundationDB release carried to its line still moves to the one tag the chart ships for it, which can be a step below what it runs, and that showed up only in the Job log. Kubernetes and KubernetesNodes releases pointing the removed image overrides at a mirror were not listed at all. Both now go into the needs-attention record next to the OpenSearch ones. The migrations image carries no charts, so the tag map is repeated in the migration, and the bats suite fails when it drifts from the chart's versions.yaml. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
033bec74fb |
fix(platform): keep one bad release from stopping migration 60
spec.values on a HelmRelease takes any shape, so a release holding a scalar there, or a cluster.version the apiserver stored as a number, crashed jq and stopped the carry-over for the whole management cluster: every healthy FoundationDB release then went into the upgrade without its version. Those releases are now reported and recorded like any other value that cannot be carried over. The same record covers OpenSearch releases that still set images.opensearch. Nothing can be carried over for them, so what the operator needs is the list. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
f3058d2353 |
fix(platform): guard the FoundationDB version carry-over on all values
Migration 60 tested only cluster.version before adding version, so a version set on the release between the scan and the patch would have been overwritten. The patch now tests the whole values object it read. resourceVersion is not used for this: helm-controller keeps writing the release status, and a mismatch there would abort the platform upgrade for no change to the values. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
930051280b |
fix(platform): carry FoundationDB cluster.version over to version
The foundationdb chart no longer reads cluster.version, so a cluster that set it to 7.1 or 7.4 would move to the v7.3 default on the first reconcile after the upgrade: a full-bounce upgrade from 7.1, or a downgrade from 7.4. Migration 60 sets version to the release line of each cluster.version it finds. It keeps cluster.version, because the old chart still renders the release between the migration and the chart upgrade and would fall back to its own default without it. A value outside the supported lines is recorded on the cozystack-version ConfigMap and does not block the upgrade. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
873bd6712d |
fix(platform): stop carrying images.kubectl into adopted node pools
Migration 56 copied the parent cluster's kubectl image override into each kubernetes-nodes release it creates. Neither chart reads an images map any more, so the copy only planted an inert key in the pool values. Migration 56 has not shipped in a stable release, so no cluster has run the old mapping outside a pre-release. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Myasnikov Daniil
|
b83cc087cc |
docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the source tree, where no operator looks. The website carries them now, so a second copy here only lets the two drift again: the Backup Classes page had already fallen behind its original. docs/release.md, docs/deprecating-applications.md and the two SeaweedFS runbooks in docs/operations stay. A migration, a bats suite, a workflow or AGENTS.md reads each of them by path, which is the test for keeping a document in this repository. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f3486f4e3a |
docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed them and left in the source tree, where no operator looks. They are on the website, so a second copy here only drifts: edits kept landing in docs/ after the website copy was made. docs/vm-external-vlan.md is split rather than moved. Its section on staging the reference CNI plugins is a contract that hack/cni-plugins-staging-contract.bats asserts against the multus Dockerfile, the daemonset manifest and the platform values, so that half is package documentation and goes to packages/system/multus/README.md, which the suite now reads. The operator half goes to the website. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
1f74ad4ee7 |
fix(multus): drop the CPU limit and make resources configurable (#4830)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The multus daemon was too small to recover a node after a CNI outage, and its limits could not be raised from values. The 100m CPU limit and the 1Gi memory limit of `kube-multus` were hardcoded in the patched upstream template. On a node with about 600 dead sandboxes and 220 pods waiting, multus was throttled in every CFS period. Its calls took longer than kubelet's two-minute CRI deadline, so every call was retried and the queue only grew. With the CPU limit lifted it OOM-looped at 1Gi. With 2 CPU and 4Gi it cleared the backlog in about three minutes. Now the resources come from a new `resources` value. The default keeps the same requests (100m CPU, 100Mi memory), has no CPU limit, and raises the memory limit to 4Gi. The memory limit stays on purpose. Without it the pod cgroup has no `memory.max`, and multus becomes a candidate for the Talos userspace OOM handler again. The chart now refuses to render when an override removes the memory limit or the memory request. Without a request Kubernetes uses the limit as the request, so a node with less allocatable memory than the limit would be left without multus. The change is in `patches/customize-deployment.patch`, so `make update` keeps it. I applied the patch to upstream v4.3.0 and got the committed template, except the image references. The platform reverts a direct edit of the child `cozystack.multus` Package, so a running cluster sets the value through the new `networking.multus.resources` key of the platform Package. The key is passed on only when set, and a value that is not a map stops the render. The system-defaults LimitRange in the namespace sets memory only, so it does not bring a CPU limit back. Closes #4829 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not a UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/747 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff adds a platform value, so the hand-written key table on the website needs it. The follow-up PR there is a draft until this one merges. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(multus): the kube-multus container no longer has a CPU limit, and its memory limit is raised from 1Gi to 4Gi, so a node with hundreds of pods can recover after a CNI outage. Its resources can be overridden through the new `networking.multus.resources` platform value. An override that removes the memory limit or the memory request now fails the render. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional CNI plugin staging during Multus startup, configurable through chart values. Staging skips when the plugin directory is unavailable and reports individual copy failures while continuing. * Made Multus resource settings configurable, with defaults of 100m CPU and 100Mi memory requests and a 4Gi memory limit. Rendering requires memory requests and limits. * Updated the Multus deployment to use the `cozy-multus` namespace and DaemonSet name, and Cilium as the master CNI configuration. * **Bug Fixes** * Added a 128Mi memory limit to the CNI installation step. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
938abdc4d2 |
fix(platform)!: refuse a networking variant change on a live cluster (#4790)
## What this PR does Changing the platform variant on a running cluster can delete its CNI, and nothing stopped that before. A new platform variant (`spec.variant` of the `cozystack.cozystack-platform` Package) can give the `cozystack.networking` Package a different variant, and the operator then deletes every HelmRelease of that Package the new variant does not install. Moving from `isp-full` to `isp-hosted` deletes Cilium and Kube-OVN, so every pod loses networking. Moving from `isp-full` to `isp-slim` deletes Kube-OVN. With this PR the platform render looks up the live `cozystack.networking` Package. If its variant differs from the one about to be rendered, the render fails, and the error names the value that lets it through: `networking.acknowledgeVariantChange: "<new variant>"`. I made the acknowledgement a variant name instead of a boolean like `publishing.proxyProtocolAcknowledgeUnclean`. A leftover `true` would let every later switch through unchecked, while a variant name covers only the switch it names. The guard does nothing when there is no live Package to compare against: `helm template`, a client-side dry-run, or a first install. It is also skipped when `cozystack.networking` is listed in `bundles.disabledPackages`, because the platform does not render the Package then. It lives in the shared Package helper and not in `bundles/system.yaml`, so it covers both places that emit the networking Package, including the `noop` call in the `isp-hosted` branch. The other Packages whose variant follows the bundle, `cozystack-scheduler` and `tenant-application`, keep the same release names in every variant, so changing their variant deletes nothing and they need no guard. The new helm-unittest cases cover the refused switches, an unchanged variant, a matching and a mismatching acknowledgement, a bare boolean, `disabledPackages`, an offline render, and a live cluster with no networking Package yet. A cluster where someone set the Helm-owned networking Package to `cilium-kilo` by hand now fails its platform upgrade, and the error points it at `bundles.disabledPackages`. Acknowledging does not help there, because the render writes the bundle's variant over `cilium-kilo`. helm-controller v1.5.0 installs a release with server-side apply by default, which takes over every rendered field. On a release it installed, the upgrade replaces the variant and the operator deletes Kilo. A release first installed with Helm 3 keeps the apply method recorded in its history, so there the outcome depends on that history. Known gap, left for a follow-up: the guard fires on any variant change, including Talos to generic (`kubeovn-cilium` to `kubeovn-cilium-generic`). Those pairs share release names, so nothing is deleted and only values change. Refusing is safe, and the error does not claim a deletion for such a switch. Closes #4596 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories This PR adds a key to `packages/core/platform/values.yaml`, and the website keeps a hand-written table of those keys in `operations/configuration/platform-package.md`. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/746 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): the platform render now refuses to change the variant of a live cozystack.networking Package, because the operator would then delete the CNI HelmReleases the new variant does not install (isp-full to isp-hosted removes Cilium and Kube-OVN). To go ahead with such a change, set networking.acknowledgeVariantChange to the new networking variant named in the error. A cluster whose cozystack.networking variant was set by hand (for example cilium-kilo) needs cozystack.networking in bundles.disabledPackages before the upgrade, otherwise the platform render refuses. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Changing the networking variant on a running cluster now requires explicit confirmation naming the requested variant. A change without a matching confirmation is blocked. * The confirmation applies only to the specified switch and must be updated for a later variant change. If the running package has no recorded variant, it is treated as using the default variant. * New installations without an existing networking package are unaffected, and networking can still be disabled through its package setting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
647407545d |
fix(backupstrategy-controller): run backup clients on a package-owned image
The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the migration runner, and whose pins were stamped from packages/core/platform so every migration change moved the backup clients too. Build images/backup-http-client (alpine + curl + jq + ca-certificates, non-root) from this package's Makefile and stamp both client keys from it. The cross-package stamp in packages/core/platform/Makefile and the platform-migrations-specific check in hack/image-pin-consistency.bats go away; the general one-digest-per-repository check still covers the two keys sharing one image. Fixes #3420 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Aleksei Sviridkin
|
ddd00f8639 |
feat(platform): pass multus resources from networking.multus.resources
The multus chart takes its resources from values, but the platform renders the cozystack.multus Package with stageCniPlugins alone, and an edit to that child Package is reverted by the platform reconcile. A running cluster therefore had no way to raise the multus limits. networking.multus.resources is passed to the Package only when set, so an unset key leaves the chart default in charge. A value that is not a map stops the render here; the multus chart keeps refusing a render without a memory limit or request. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1245b32a32 |
fix(platform)!: refuse a networking variant change on a live cluster
Changing the variant of the cozystack.cozystack-platform Package on a running cluster changes bundles.system.variant and can re-render the cozystack.networking Package with another variant. The operator then deletes every HelmRelease of that Package the new variant does not install, so isp-full -> isp-hosted removes Cilium and Kube-OVN and the cluster loses its CNI; isp-full -> isp-slim removes Kube-OVN. The platform render now looks up the live Package and fails when its variant differs from the one about to be rendered, unless networking.acknowledgeVariantChange names the new variant. The acknowledgement is a variant name rather than a boolean so that a value left behind after one switch does not cover a later one. The check is inert where lookup returns nothing (helm template, client-side dry-run, first install) and skipped when cozystack.networking is in bundles.disabledPackages, since the platform then does not render it. A live variant set by hand, such as cilium-kilo, is pointed at bundles.disabledPackages rather than at the acknowledgement: the acknowledged render would write the bundle's variant over it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a2e10312de |
fix(platform): stop treating a failed ConfigMap read as a fresh cluster (#4792)
## What this PR does A single failed API read at the start of the platform migration Job could skip every pending migration and still end green. `run-migrations.sh` checked whether the `cozystack-version` ConfigMap exists with a silenced `kubectl get` and looked only at the exit code. A timeout, a dropped connection or an apiserver restart during the upgrade all counted as "no ConfigMap", so the script stamped the target version and exited 0. The hook Job is rendered only when that ConfigMap already exists, so in practice an error was the only way into this branch. Now the script creates the ConfigMap only when it is really missing. The check uses `--ignore-not-found --output name`: kubectl drops NotFound and nothing else, so empty output means a genuine NotFound. Any other error keeps a non-zero exit, `set -e` aborts the script, and the Job retries the pod. The rest of the script already works this way: the read of the stamped version aborts on error, and the stamp runs under `pipefail`. The tests cover both branches. A new case in `hack/run-migrations.bats` makes the existence check fail with a connection error and expects a non-zero exit, no migration run and the stamp left where it was. It fails against the old check. The fake kubectl returns empty output for a missing ConfigMap only when `--ignore-not-found` is passed, as real kubectl does, so dropping the flag breaks the existing "no ConfigMap" case. No migration is added, so `targetVersion` stays. The script ships in the migrations image, and the release build restamps that image's digest as usual. Several older migrations have the same problem in a smaller form. They check for an object with `kubectl get ... >/dev/null 2>&1` and skip their own work when the check fails, for example migrations 31, 32, 36, 48, 50 and 51. A transient error there skips one migration's work, not all of them. Editing migrations that already shipped has its own risk, so I'd track that separately. Closes #4579 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): a transient API error while the platform migration Job checks for the cozystack-version ConfigMap now fails the pod so the Job retries, instead of stamping the target version and silently skipping every pending migration. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Migration checks now distinguish a missing version record from an error while checking for it. When the check fails, the pod stops instead of treating the record as missing and proceeding with migrations. * Added coverage for TLS-handshake timeouts during the check, confirming that migrations do not run and the stamped version remains unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
87a64071a7 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does
fixes #3740.
on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.
**what changes**
- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.
**existing clusters**
a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.
1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.
running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.
### Validation
- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at
|
||
|
Aleksei Sviridkin
|
0a2bb5ef98 |
fix(cert-manager): enable the Gateway API solver in the host cert-manager (#4039)
## What this PR does With the platform Gateway enabled, the ClusterIssuers in `cert-manager-issuers` and the per-tenant Issuers rendered for `TenantGateway` carry an `http01.gatewayHTTPRoute` solver, but the controller refuses it: the vendored chart never turns Gateway API support on, so every HTTP-01 challenge behind the Gateway fails with `gateway api is not enabled` and no certificate is issued. This passes a `ControllerConfiguration` with `enableGatewayAPI: true` to the host cert-manager through the component values of its platform Package. The host ships the Gateway API CRDs on every variant and orders cert-manager after them, so the flag is safe there. It is deliberately not a default of the `system/cert-manager` chart. The same chart is installed into tenant Kubernetes clusters, where the CRDs are an opt-in addon, and cert-manager v1.20 exits at startup when Gateway API support is on and the CRDs are missing. The tenant HelmRelease in `packages/apps/kubernetes` already passes the same `ControllerConfiguration` when its `gatewayAPI` addon is enabled, so both clusters now opt in from the layer that knows whether the CRDs are there, and the chart itself stays neutral. The new tests pin both halves: the host Package carries the flag and the preconditions that make it safe, and the chart wires `config` through to the controller while shipping none by default. Closes #3725 ### Screenshots Not applicable, no UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(cert-manager): enable the Gateway API solver in the host cert-manager, so ACME HTTP-01 challenges succeed when the platform Gateway is enabled ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Supported system bundles can now use cert-manager’s Gateway API HTTPRoute solver to issue certificates. - Hosted deployments include the required Gateway API custom resource definitions, installed before cert-manager. - Gateway API solver support remains opt-in in environments that do not provide the required definitions, avoiding unintended changes to certificate management there. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
76b9e55102 |
fix(cert-manager): enable the Gateway API solver in the controller
With the platform Gateway enabled, the ClusterIssuers in cert-manager-issuers and the per-tenant Issuers rendered for TenantGateway carry an http01.gatewayHTTPRoute solver, but the controller only honours that solver when Gateway API support is switched on, and the vendored chart leaves it off. Every HTTP-01 challenge behind the Gateway then fails with "gateway api is not enabled" and no ACME certificate is issued. Pass a ControllerConfiguration with enableGatewayAPI to the host cert-manager through the component values of its platform Package. The host always ships the Gateway API CRDs and orders cert-manager after them, so the flag is safe on every variant. It is not a default of the system/cert-manager chart: the same chart is installed into tenant clusters, where the CRDs are an opt-in addon and a controller started with Gateway API support and no CRDs exits at startup. The tenant side keeps opting in on its own. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM |
||
|
Aleksei Sviridkin
|
cbbb69afc3 |
fix(platform): stop treating a failed ConfigMap read as a fresh cluster
run-migrations.sh decided whether cozystack-version exists by the exit status of a silenced kubectl get, so a timeout or a dropped connection counted as "no ConfigMap": the runner stamped the target version and exited 0, and every pending migration was skipped for good. The hook Job only renders when the ConfigMap already exists, so in practice an error was the only way to reach that branch. --ignore-not-found suppresses NotFound alone; any other error keeps its non-zero status and now fails the pod, so the Job retries the read. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
d61fed46cc |
feat(local-storage): add opt-in OpenEBS LocalPV hostpath package
LINSTOR needs the DRBD and ZFS kernel modules, so a cluster without them has no dynamic PV provisioning. OpenEBS LocalPV hostpath needs no kernel modules and also serves deliberately node-local workloads. The package is opt-in so it cannot change an existing cluster. Its class is named cozystack-local-hostpath so it cannot collide with an existing class name, and it is not the default so PVCs that omit storageClassName keep their current binding. The vendored RBAC is patched down to what volume reconciliation calls; helper Pods, Leases and analytics state stay in the install namespace. Images stay on docker.io, pinned by index digest. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
974aaa0bac |
test(hack): regenerate the selective install fixture
Two optional Cluster API providers, Proxmox and in-cluster IPAM, were added to the platform sources after the fixture was generated. The drift test compares the fixture with the current disable list, so it failed until they were added. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f46a9797b8 |
fix(hack): keep platform services in selective installs
Dropping every package that no suite owns also drops services the rest of the platform relies on: multus for secondary networks, the cozystack and LINSTOR schedulers, flux-plunger, cozy-proxy and ouroboros, whose hairpin NAT the Kubernetes suites assert. Nothing waits for them, so no PackageSource edge reaches them, and adding one would also change reverse test selection. The remaining packages stay uninstalled on purpose, also when test selection answers with every suite: nothing in the run looks at them, so installing them only adds a readiness wait. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
59555c895e |
fix(hack): retain controllers and APIs for selective installs
Tenant HelmReleases need a shard reconciler, and the root SeaweedFS release creates COSI resources even when no backup suite is selected. Without those controllers, a fresh reduced install cannot become ready. The Kubernetes suites create worker pools through KubernetesNodes, and MongoDB renders an operator-owned CR. Selecting only the cluster API or application chart drops resources those suites require. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
361893e6f4 |
fix(hack): retain backup APIs in selective installs
Backup roundtrip suites create Bucket and backup resources that the application graph does not require. A graph-only closure therefore omits the APIs those suites need on a fresh install. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
610578be16 |
test(hack): verify selective install contract
A missing suite owner or a discarded discovery failure can leave a reduced install without the APIs its tests exercise. Graph-only checks cannot establish that the chart preserves the requested package set. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Yan Bondarenko
|
dc362fa591 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent still takes its router IP and the Gateway API Ingress IP from its allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which the documented installs slice from the same range kube-ovn allocates pods from, so kube-ovn sooner or later hands a pod an address Cilium already holds. Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a /29 per node from the last /21 of networking.podCIDR, and kube-ovn excludes that /21 through --default-exclude-ips. The block stays inside the pod CIDR on purpose. Envoy reaches Gateway API backends from the node's Ingress IP, and kube-ovn masquerades host-originated traffic only when the source lies in one of its subnets, so a block outside them such as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes. A post-upgrade hook moves existing clusters. It first checks that the block has a /29 for every node and lies inside kube-ovn's default subnet, and stops without changing anything otherwise. It adds the block and every node's current Ingress IP to excludeIps of that subnet, because the agent keeps a former Ingress IP in its ipcache as reserved:ingress. It then waits until kube-ovn stops offering the block and stops if an address in it is still allocated. kube-ovn leaves excluded addresses out of the subnet status, so that check reads its IP objects. Then, one node at a time, it clears the inherited /24 from the CiliumNode, waits for the operator to allocate from the block and restarts the agent. networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn variants, and the render fails otherwise. A /21 cut into /29s also caps these variants at 256 nodes. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
bcda5c893a |
feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves the platform to declarative volume hotplug. It changes how every volume hotplug on the platform works, so the title carries `!`. Closes #4032. This PR is stacked on #4451 and has to merge after it. #4451 adds the kubevirt-csi flag this PR flips. The replacement gate is already on in KubeVirt 1.9, and the old gate is the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at v1.9.0, and Beta gates are on by default. virt-controller skips `handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so dropping the old gate is what switches the new one on. The kubevirt-csi fallback goes off in the same change. With the old gate off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments next to the gate and the flag point at each other. This needs the rebuilt kubevirt-csi image. A driver built before #4451 does not know the flag and exits on it at startup. The image is stamped at release preparation and ships in the same release as this change, so only a `make apply` from main before then would crash-loop the CSI controller. What changes at runtime on an upgrade: - The VM-level `addvolume` and `removevolume` calls now edit `VM.spec` directly. virt-controller then makes the VMI hotplug volumes match it, instead of going through `VM.status.volumeRequests`. - Volumes hot-plugged through the VM subresource are in `VM.spec`, so they stay attached. - During a tenant teardown, when the VM goes before its VMI, the driver no longer calls the VMI subresource. It waits for the VMI status to drop the volume, and the garbage collector deletes the VMI with its VM. Upstream waits up to two minutes. If the VMI is still there after that, the unpublish fails and is retried. A new platform migration keeps old VMI-only volumes attached. On its first pass virt-controller removes every hotpluggable VMI volume that is missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at the VMI level (the Role then granted `virtualmachineinstances/addvolume`), and live migration carries such a VMI across KubeVirt updates. Without help, a tenant node not recreated since then would lose a mounted disk on upgrade. Migration 59 copies each such volume and its disk from the VMI into the VM template before anything else changes. Adding a hotplug volume to the VM spec is a valid live update in KubeVirt, so no VM needs a restart. It runs as the platform chart's pre-upgrade hook, before the upgrade re-points `cozystack-packages`. The KubeVirt CR and the tenant kubernetes charts both render from that source, so the gate and the flag cannot change before the migration has finished. It only touches VMs that need it, logs each one, and a second run changes nothing. The migration also makes ephemeral hotplugs permanent. A volume a user added at the VMI level, for example with `virtctl addvolume` without `--persist` from an older virtctl, has the same shape as kubevirt-csi's old attachments, because virt-api builds both the same way. There is nothing reliable to tell them apart by, so both are copied. Leaving the user's volume out would not keep it ephemeral: the first reconcile would unplug it from the running VM. After the upgrade it is part of `VM.spec` and attaches again on a restart, and `virtctl removevolume` drops it. Ephemeral hotplug stops working on VMs. With the old gate off, virt-api refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns (`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and removed through the VM from then on. A VMI without a VM is not affected. Only running VMIs are copied. virt-controller reconciles no other VMI, and a stopped VM builds its next VMI from `VM.spec`, so copying from a stopped one would bring back volumes the restart was going to drop. Migrations run only on the isp-full, isp-full-generic and isp-hosted variants. KubeVirt comes from the platform only on isp-full and isp-full-generic, so those are covered. On the default variant, where packages are installed by hand, nothing runs the migration. This optional check lists the VMIs and volumes the migration would copy. It uses the same rule. On the default variant, run it before upgrading and recreate each tenant node it lists: ```bash kubectl get vm,vmi --all-namespaces --output json | jq --raw-output ' ([.items[] | select(.kind == "VirtualMachine") | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}] | from_entries) as $vm | .items[] | select(.kind == "VirtualMachineInstance") | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner | "\(.metadata.namespace)/\($owner)" as $key | select(.status.phase == "Running" and $owner != null and $vm[$key] != null) | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true) | .name as $n | select($vm[$key] | index($n) | not) | "\($key) \($n)"' ``` Empty output means nothing is affected. I checked the query against the migration's test fixtures only. Neither the query nor the migration has run on a live cluster. #4287 also adds a migration 59. If it merges first, this one moves to 60: the file, the version it stamps, `migrations.targetVersion`, the bats file (its `run_migration` argument and expected `STAMP`) and `hack/testdata/migration-59-vmi-hotplug`. A VM with a pending hotplug request at upgrade time may also get stuck. virt-controller still processes `VM.status.volumeRequests` before the declarative handler, and it applies them through the VMI subresource, which virt-api rejects once the old gate is off. The sync then fails before the VM spec is updated, so nothing clears the request, and every later sync of that VM fails the same way. A retry through the VM subresource edits the spec and clears it, which covers kubevirt-csi's own requests. A one-shot request stays stuck until the VM restarts. I read this from the v1.9.0 code and haven't reproduced it. Waiting until this lists nothing before upgrading narrows the window, but a request made during the rollout can still hit it: ```bash kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name' ``` The two halves live in different HelmReleases, so there is a short window during the rollout where the driver and the gate disagree. In that window an unpublish fails and is retried until both are updated. A tenant cluster whose `kubernetes` HelmRelease is suspended or failing keeps its old driver for longer. Drivers from before July 2025 attach through the VMI subresource, so on such a tenant every attach is refused until its release reconciles. Such a driver can also attach a volume after the migration has run and before the gate flips. The migration doesn't see that volume, and the first declarative reconcile unplugs it from the running VM. The migration lists the VMs once and patches each one with a `resourceVersion` test, so a VM that changes in between fails the hook Job. The Job has `backoffLimit: 3`, and each retry lists again and skips the VMs already patched. Re-reading only the conflicting VM would avoid the retry; I left that out. The new gate is listed in the CR even though it is on by default. With the old gate gone it is the only hotplug path, and the chart refuses a render that disables it. I found no other user of `HotplugVolumes` in the platform or in the downstream repositories. The VMI-level fallback may have been failing already. The kcsi Role has no rule for the VMI-level `removevolume` (#4452), so that call likely got 403 with or without the gate. I haven't checked that on a cluster. `hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left alone (a VMI that is not Running among them), a second run, a cluster without KubeVirt and the fail-closed paths. The helm-unittests for the KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in `disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The CSI test pins the flag at false, and the `make update` idempotency test still passes. None of that exercises hotplug itself. The real test is e2e with volume attach and detach on a tenant cluster, including a teardown. No CI lane upgrades from the previous release yet, so an upgrade with a live VMI-only volume is untested; #4463 tracks that test. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Declarative hotplug volumes are enabled by default for KubeVirt. * Running virtual machines keep eligible hotplugged volumes and disks when their configuration is migrated. * **Bug Fixes** * The migration avoids adding duplicate volumes or disks and does not mark the migration complete if required operations fail. * The Kubernetes CSI driver no longer uses the VMI hotplug fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
e63b9e439c |
fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and burst was the receiver's 4 MiB cap. A request grows on its way to the limiter: the sampler adds a tracestate to every span it keeps, and the limiter counts a request as the collector re-encodes it, which writes the empty value of every attribute the wire may omit, so a valid request of spans with empty-valued attributes is counted 1.4x. A request the receiver admitted near its cap was therefore refused on every retry. The limiter now runs right after memory_limiter, so it counts the request before sampling. The receiver caps a request at 3 MiB and burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would also be what each pod start lets through above the rate. The rate is counted before sampling, so the store keeps only the sampled share of it. The texts say that a request above the cap gets a 400, which senders do not retry, and that each collector rollout a tenant causes lets that burst through again on every pod. The replica bound that keeps that allowance small now applies only to a central tenant's collector, whose render fails above ten replicas: a collector writing to the tenant's own store has no rate, and the replica counts it shipped with stay valid. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
3497855e15 |
fix(platform): read the tracing rate only as plain decimal digits
Sprig's int64 reads a string with a leading zero as octal and turns a parse error into 0, so "0100" would reach the collectors as 64, and "08" or digits beyond int64 would leave the default in place without a word. A float beyond int64 converts differently on each CPU architecture, and the collector adds its replica count to the rate before splitting it. A string rate must now be plain decimal digits, and any rate above 1 TiB/s fails the render; the collector also falls back to the default on digits it cannot hold rather than start with a rate of zero, which the processor refuses. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
a0aea139fc |
fix(platform): refuse a malformed shared-central tracing rate
A rate written as a quantity or a fraction ("1Mi", "100KiB", 1.5) was
read as zero, so the platform wrote no key and every collector kept
the default without a word. The platform render now fails on anything
but a whole number of bytes per second. Helm reads a number from
values as a float64, which prints a million as 1e+06, so a whole
number is checked as a number and written as int64 digits.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
Alexey Artamonov
|
fc0db49187 |
fix(monitoring): bound what a central tenant writes above its rate
Each collector pod starts with a full token bucket, and burst was at least 64 MiB, the largest request vtinsert accepts. A tenant controls when its collector pods start, through any Monitoring change or by deleting them, and how many there are, so it could write tens of mebibytes above its rate per pod start and gigabytes per rollout of many replicas, which the two-day bound does not survive. The limiter only sees requests the receiver admits, so in central mode the receiver now caps them at 4 MiB, decompressed for HTTP, and burst drops to that floor; a pod start lets at most 4 MiB through above the rate. The replica count is capped at 10. The texts now say what the default rate guarantees: one tenant writes at most a default store's disk cap over two days counted uncompressed, rather than that it cannot fill it, and they name the per-pod-start allowance and that a sender pinned to one replica gets that replica's share. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
f8deb8e171 |
fix(platform): write the tracing rate only when it is positive
apps.yaml wrote the rate for any non-zero value, so a negative one reached every tenant's cozystack-values Secret and upgraded every release in every tenant, while the collectors ignored it and kept the default. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
177b207a27 |
feat(monitoring): hold each shared-central tenant to a write rate
The design has the per-tenant collector rate-limit its tenant, so that a noisy or hostile tenant cannot exhaust a shared backend. The disk cap alone does not do that: VictoriaTraces always keeps the last two days, so a tenant writing faster than the cap allows over two days fills the disk for every tenant on the store. The platform sets the rate with monitoring.tracingCentralTenantBytesPerSecond, and it reaches every tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each monitoring HelmRelease reads, so Flux upgrades them all; a rate read with lookup at render time would never reach a tenant whose release is already in sync. The platform writes the key only when it is set, since every release in every tenant reads that Secret. Without it the collector holds the tenant to the rate of a default 10Gi store. Each central tenant's collector splits the rate across its replicas and refuses data above it before the batch processor, so the sender sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which the OTLP specification makes final, so central tenants should export over HTTP. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
7bf6906a80 |
fix(platform): keep the legacy etcd-operator stopped after etcd adoption (#4738)
## What this PR does Migration 50 scales the legacy etcd-operator to 0, adopts every legacy `etcd.aenix.io` cluster with `etcd-migrate`, and then scales the operator back to 1. At that point the Deployment still runs the legacy operator, and after a successful adoption it has nothing left to manage. Bringing it back undoes part of the adoption. Observed on a production cluster upgraded from 1.4.3 to 1.6.4. The API audit log and the Helm history of a tenant's `etcd` release show this sequence, in UTC on the same day. - 13:21:12 migration 50 scales `cozy-etcd-operator/etcd-operator-controller-manager` to 0. - 13:22:59 `etcd-migrate` updates `etcd-headless`, pointing its owner references at the adopted EtcdMembers, and orphan-deletes the legacy EtcdCluster and StatefulSet. - 13:23:04 migration 50 scales the operator back to 1. It is still the legacy operator, the v1alpha2 one replaced it only at 13:40. - 13:23:42 Flux upgrades the tenant's `etcd` release with the old chart (Helm revision 87), because the new chart artifact appeared only at 13:24:40. The old chart re-creates the legacy EtcdCluster at 13:23:43. - 13:23:43 to 13:23:45 the legacy operator reconciles it, updates `etcd-headless` twice and re-creates the legacy StatefulSet `etcd`. - 13:36:49 Flux upgrades the release to the new chart (revision 88), which prunes the legacy EtcdCluster. - 13:36:50 the garbage collector deletes `etcd-headless` and the re-created StatefulSet. The same happened in all five tenants that run an etcd. The pods survived because EtcdMembers own them, but they keep `spec.subdomain: etcd-headless`, so their `etcd-<i>.etcd-headless.<ns>.svc` names stopped resolving. From then on every member logged `no such host` for its peers, the v1alpha2 EtcdCluster had no status conditions, and the `etcd-defrag` CronJob failed every hour with `context deadline exceeded`. The transitional `etcd-headless` Service the etcd chart ships for this window (#3270) does not cover it. Helm patched the existing Service 202 ms before the garbage collector removed it, and did not create it again because it was already part of the release. With this change the operator stays at 0 after a successful adoption. The etcd-operator HelmRelease replaces the Deployment with the v1alpha2 operator later in the same upgrade, and no later migration waits for the operator. A failed or interrupted adoption still restores it to 1 through the existing EXIT trap, so a failed run does not leave the tenant etcds without an operator. If that HelmRelease cannot reconcile, the adopted clusters wait for it with or without this change, because the legacy operator does not reconcile v1alpha2 EtcdClusters. Clusters already upgraded with this bug can be repaired without restarting anything. Creating the Service from the tenant's own `etcd` release is enough. ``` helm -n <ns> get manifest etcd | awk 'BEGIN{RS="\n---\n"} /\nkind: Service\n/ && /\n name: etcd-headless\n/' | kubectl -n <ns> create -f - ``` On the cluster above the v1alpha2 EtcdCluster reported `Available` with reason `QuorumHealthy` within 11 seconds of the Service being created, in every tenant, and the members stopped logging `no such host`. ### Validation - `hack/cozytest.sh hack/migration-50-etcd-adopt.bats` passes all 29 tests. A new test asserts that a successful adoption scales the operator to 0 and never back to 1. Against the current script it fails with `FAIL: the legacy operator was scaled back up after a successful adoption`. - The existing test for a failed adoption still passes, so the EXIT trap still restores the operator to 1. - The new test checks the absence with `if grep ...; then echo "FAIL: ..."; false; fi`, the form the other suites use. A bare `! grep` assertion cannot fail a test under `hack/cozytest.sh`, so it could not have caught this. The other negated assertions in this file had the same weakness, and #3790 has rewritten them on main. With all of them made able to fail in a scratch copy, the suite still passes against this change. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. It changes a migration script and its bats test. No package, values, schema, CRD, make target or `hack/` layout changes, so no entry matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): keep the legacy etcd-operator stopped after migration 50 adopts the legacy etcd clusters. Scaling it back up let it reconcile the legacy EtcdCluster that Flux re-created from a not yet upgraded etcd chart, re-create the legacy StatefulSet and take the legacy etcd-headless Service away from the adopted members, which the garbage collector then deleted. The adopted members lost their DNS names, the EtcdCluster never reported a status and etcd defrag failed every hour. Clusters already upgraded can recreate the Service from the etcd release manifest. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Migration Updates** * After successful etcd adoption, the legacy operator remains scaled to zero replicas for replacement by the etcd operator; it is no longer scaled back up before the migration is stamped. * If adoption fails or exits before completion, the legacy operator is restored to one replica. * The dry-run message now describes the zero-replica outcome of successful adoption. * Successful adoption continues to record the migration version stamp. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
50dec5d46a |
fix(platform): resume a retried migration Job from the stamped version (#4583)
## What this PR does A retried pod of the platform migration Job no longer runs migrations that already finished in the same upgrade. The Job gets `CURRENT_VERSION` once, at render time, and a failed pod is replaced by one with the same environment. By then every finished migration has stamped the `cozystack-version` ConfigMap, but the new pod ignored that and started from the rendered version again. `run-migrations.sh` now starts from the version stamped in the ConfigMap. If reading it fails, the pod fails and the Job retries it, because falling back to the rendered version is exactly the re-run this fixes. The rendered value is used only when the ConfigMap holds no integer version, which is also when the chart itself could not get a real version from it. Without that fallback a value like `v58` gave a green Job that ran nothing. The new `hack/run-migrations.bats` runs the script in the migrations image's alpine base against a fake kubectl that keeps the stamp in a file, so one pod's stamps are what the next pod reads. It needs docker, like the other migration suites. I also found that the ConfigMap existence check at the top of the script treats any API error as a missing ConfigMap and skips every pending migration. That is older than this change and is filed as #4579. Closes #4555 ### Screenshots ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): a retried platform migration Job now resumes from the version stamped in the cozystack-version ConfigMap instead of re-running migrations that already finished in the same upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Migration startup now resumes from a valid stamped version, avoiding reruns of migrations that have already completed. * Empty or non-numeric stamps fall back to the rendered version. Out-of-range stamps or failed stamp reads stop migrations before they run. * If the migration ConfigMap is missing, migrations do not run and the target version is stamped. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yan Bondarenko
|
6248d458ad |
fix(platform): keep the legacy etcd-operator stopped after etcd adoption
Migration 50 scales the legacy etcd-operator to 0, adopts every legacy etcd.aenix.io cluster with etcd-migrate and then scales the operator back to 1. At that point it is still the legacy operator, and it has nothing left to manage. Until a tenant's etcd HelmRelease reconciles the new chart, Flux can upgrade it with the old one, which re-creates the legacy EtcdCluster that etcd-migrate just orphan-deleted. The legacy operator then reconciles it, re-creates the legacy StatefulSet and rewrites the legacy etcd-headless Service, dropping the owner references etcd-migrate pointed at the adopted EtcdMembers. When the new chart prunes the legacy EtcdCluster, the garbage collector deletes the Service with it, and the adopted members' etcd-<i>.etcd-headless names stop resolving. After a successful adoption the operator now stays at 0 until the etcd-operator HelmRelease replaces the Deployment with the v1alpha2 operator later in the same upgrade. A failed or interrupted adoption still restores it through the EXIT trap. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
3a07885b70 |
fix(platform): resume a retried migration Job from the stamped version
The migration hook Job gets CURRENT_VERSION once, at render time, and a failed pod is retried with that same environment. Each migration that finished has stamped the cozystack-version ConfigMap by then, yet the retry started from the rendered version and ran those migrations again. Migrations are written to run once, so a re-run is not safe in general. The runner now starts from the version stamped in the ConfigMap. A failed read aborts the pod, since falling back would bring the re-run back and the Job retries anyway. The rendered value is kept only when the ConfigMap holds no integer version, which is also the case where the chart could not derive a real one from it. An all-digit stamp past the shell's integer range aborts too: its comparison fails inside an if, out of reach of set -e, and the Job would go green having run nothing, while the chart renders that stamp as 0. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
IvanHunters
|
339be8d42d |
fix(vm-default-images,kubernetes-worker-image,vm-disk): wait for golden images to import and refuse cloning a failed one (#4392)
## What this PR does Both golden image catalogs, `vm-default-images` and `kubernetes-worker-image`, went Ready before any image had imported. Each one renders a DataVolume per image with immediate bind, so every import starts at install, but neither PackageSource had `healthCheckExprs`. The HelmRelease went Ready as soon as helm applied the DataVolumes, and an image that failed or stalled stayed green. This PR makes both releases wait for the imports. Each component's `install:` block now has a DataVolume health check. `current` holds only on `Succeeded`, and `failed` holds on `Failed`. Any other phase, `Unknown` included, matches neither, and flux keeps waiting. A DataVolume with no `status` key makes the CEL evaluation error out. Flux reports that as Unknown and keeps waiting too. The other settled phases are safe to treat as not ready for these charts. `PendingPopulation` and `WaitForFirstConsumer` are where a volume waits for a consumer, and immediate bind removes that wait. `UploadReady` and `Paused` belong to upload and multi-stage sources, and both charts render only `source.http`. There is no per-component timeout, so I did not set one. The window is the operator-wide `--helmrelease-install-timeout`, 10m by default. The operator sets `RetryOnFailure` on both install and upgrade, so a download that outlasts the window is retried as an upgrade until the import finishes. Nothing uninstalls the DataVolumes in between, and the import keeps running. One image that never imports keeps its whole catalog not Ready until the image imports or its entry is removed. There is no time limit, and I kept it that way on purpose. A group/kind health check cannot tell one image from another, so it cannot be made per image. A per-component timeout would be an API change for one opt-in package. Both charts' `values.yaml` document this. Nothing on `main` lists either package in `dependsOn`, and the images that did import stay usable in the meantime. Tenant worker pools look up each golden directly and never wait on the catalog's status. The price is a helm-controller slot. The operator labels the generated HelmRelease with `cozystack.io/package`, plus `cozystack.io/privileged` when privileged. The flux-shard-operator acts only on HelmReleases that already carry `sharding.fluxcd.io/key`. So both catalogs go to the system helm-controller, which picks up releases without that key and runs with `--concurrent=5`. A DataVolume in `Failed` ends each attempt at once and costs next to nothing. An import that stalls without reaching `Failed` is the expensive case, for example an importer that keeps retrying an unreachable URL. Each attempt waits out the full window, and the next one starts 30s later. So one of those five slots stays busy about 95% of the time for as long as the image is stuck. #4145 describes the same cost on the shard controllers. The status change alone does not protect a vm-disk, because vm-disk clones from a golden without checking it. A golden still importing is fine, since CDI holds the clone until the source is populated. A golden whose import has Failed is never populated, so the clone would wait forever. vm-disk now refuses to render a new disk on a Failed golden and names the image. This matches the check the worker pools already have. A disk that already exists is exempt, so a golden that fails later does not break that disk's upgrades. The check fires only when CDI sets Failed, and at CDI v1.66.1 an import reaches Failed only on a checksum mismatch. The goldens carry no checksum, so a broken URL keeps the importer retrying and the new disk still waits, with only the catalog's not-Ready status pointing at the image. #4564 tracks that gap. A golden that does not exist at all is a separate gap, tracked in #4548. Tests pin the expressions for both catalogs. A helm-unittest case checks them on the rendered PackageSource. A Go test builds the HelmRelease spec from the shipped file and checks that they arrive with the poller wait strategy. The repo has no harness that evaluates flux CEL, so the phase behaviour above is argued from flux's evaluator and not exercised by a test. The vm-disk check has helm-unittest cases for a Failed, a Succeeded, and a still-importing golden, for a render with no cluster, and for an existing disk. Closes #4169 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(vm-default-images, kubernetes-worker-image): the golden image catalogs now report Ready only after their images have finished importing, and a failed or stalled import keeps the catalog not Ready. A vm-disk created from a golden image whose DataVolume is in phase Failed is now refused with an error naming the image, instead of waiting forever on a clone that cannot complete. ``` |
||
|
Aleksei Sviridkin
|
de570a3806 |
feat(platform): add slim installation variants (#4595)
## What this PR does This adds three minimal platform variants for small installs, such as arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos, `isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for clusters where the host provides CNI and storage. A slim variant installs only the base platform: the engine, API and dashboard, tenants, ingress and Gateway API. The two non-hosted variants also get LINSTOR. objectstorage-controller stays on too, because cozystack-controller watches BucketClaim unconditionally and crash-loops without that CRD. Everything else is opt-in through `bundles.enabledPackages`, including every paas and naas application and its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA, Multus and MetalLB. The iaas bundle is refused, so there are no VMs and no managed Kubernetes. With the stock preset a slim variant emits 20 Packages (15 for hosted), where isp-full emits 77. On the two non-hosted slim variants, networking is Cilium alone. Without VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a `CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by default. `networking.encryption` only drives Kube-OVN IPsec, so the slim variants refuse it instead of silently ignoring it. The tenant application needs its own slim variant. The default one waits on the monitoring, etcd and SeaweedFS applications, so on slim cozystack-basics and tenant-root would never become ready. tenant-rd and cozystack-basics still reference the default variant's artifacts by name. Those artifacts exist anyway, because the PackageSource builds them for every variant. The full variants don't change. I rendered isp-full, isp-full-generic and isp-hosted against main and got byte-identical Packages. The only new output there is the slim variant in the tenant-application PackageSource. Things to know before using it: - An opt-in package does not pull in its dependencies. The presets and the docs list the chains. A package enabled without its chain stays `DependenciesNotReady`. - Slim is for new installs. Switching a live isp-full cluster to isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`, the operator removes the Kube-OVN release, and running pods lose networking. The other Packages are kept by `helm.sh/resource-policy: keep`, so the switch doesn't make the cluster smaller either. - No e2e suite runs a slim variant yet. Helm unit tests cover the presets, opt-in, OIDC, the networking values and both refusals, and a dependency check over every emitted Package passes for all three presets. I installed isp-slim from a build of this branch on a fresh three-node Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready, with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From outside, the dashboard and the API answered by name with valid certificates. A LoadBalancer Service got an address from a Cilium pool and answered through L2 announcements. Pod-to-pod traffic across nodes, cluster DNS and a replicated LINSTOR volume worked. Opting in postgres-operator and postgres-application gave a Ready Postgres. That run found the BucketClaim crash-loop above, which is fixed here. The base platform used about 215m CPU and 3.2 GiB of memory, not counting the Kubernetes control plane. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#718 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [x] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: cozystack/ansible-cozystack#79 - [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: cozystack/ccp#23 - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim` platform variants, with support for PaaS and NaaS bundles. * Slim variants use Cilium networking and include a reduced set of packages by default. Additional packages and their dependencies can be enabled as needed. * **Limitations** * IaaS and the platform encryption toggle are not supported on slim variants. Cilium-native encryption is not configured. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d463e00da1 |
feat(platform): add publishing.serviceDomain for the dashboard Services tab
The dashboard has no way to show a connection hostname for a LoadBalancer Service: before 1.4.0 the old UI composed one from the Service name and the cluster's root host, and operators whose DNS publishes LoadBalancer IPs under a dedicated zone still need that. Add the suffix as a platform value and propagate it into the cozystack-values map as service-domain, next to cluster-domain and branding, so the dashboard chart can read it. The value is display-only and opt-in: empty (the default) means the console shows no hostname, because nothing in the platform publishes DNS for a guessed name. Leading and trailing dots are stripped here, the one place every consumer reads from, so an operator's ".svc.example.org." cannot render as "demo..svc.example.org."; the value is coerced to a string first because an unquoted numeric scalar would otherwise abort the whole platform render. The helm-unittest suite pins the empty default, a configured value landing in the Secret, the dot trimming and the numeric case. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
506dc7a4a7 |
feat(platform): add isp-slim, isp-slim-generic and isp-hosted-slim variants
Small installs such as arm64 labs had to take the whole isp-full set: KubeVirt, backups, monitoring, etcd, SeaweedFS and every paas/naas operator. The slim variants keep only the engine, tenants, ingress, gateway, LINSTOR storage and networking, and turn everything else into opt-in packages listed in bundles.enabledPackages. Full variants render the same Packages as before. Networking on isp-slim and isp-slim-generic is Cilium alone: without VMs nothing needs KubeOVN, and Cilium L2 announcements take the place of MetalLB, which stays available as an opt-in package. KubeOVN IPsec is what networking.encryption drives, so the slim variants refuse it rather than ignore it. isp-hosted-slim keeps the noop datapath. tenant-application gains a slim variant whose dependsOn drops the monitoring, etcd and seaweedfs applications; the default variant waits on them, which would leave cozystack-basics and tenant-root never Ready. The iaas bundle stays limited to the isp-full variants. objectstorage-controller stays on everywhere: cozystack-controller watches BucketClaim unconditionally, so without that CRD its manager never syncs its caches and crash-loops. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
b8bc0dfd5f |
fix(platform): make mongodb-application depend on its operator
The mongodb chart renders kind: PerconaServerMongoDB from psmdb.percona.com/v1, but cozystack.mongodb-application depended only on networking and cozystack-engine. So mongodb-rd registers the ApplicationDefinition as soon as the engine is up, and a tenant can create a MongoDB before the operator exists; its HelmRelease then fails with no matches for kind "PerconaServerMongoDB". This is the same defect #3817 fixed for etcd. Every other operator-backed app here already carries the edge. It also closes a test-selection gap: without it cozystack.mongodb-operator reaches no runnable suite, so every change to the operator escalates to a run of all suites instead of selecting mongodb. select-install.sh --validate reports the graph still has no cycle. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
a09eded9ec |
fix(kubernetes-worker-image): wait for goldens to import before Ready
The Talos worker image catalog renders its goldens the same way the VM image collection does: DataVolumes with immediate bind whose import starts at install. With no health check the release went Ready as soon as helm applied them, so a golden that never imported left the catalog green. Gate it on the DataVolume phase with the same expressions. Worker pools read each golden directly and do not wait on the catalog status, so the gate changes what the status reports and delays no pool. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
51ac18776e |
fix(vm-default-images): wait for golden images to import before Ready
Every image is a DataVolume with immediate bind, so its import starts at install, but nothing waited for it: the HelmRelease went Ready as soon as helm applied the DataVolumes. An image that failed or stalled left the component green, and the failure surfaced later on a vm-disk cloning from it, far from the cause. Gate the release on the DataVolume phase. Only Succeeded counts as ready, and a Failed phase, when CDI sets one, ends the wait at once. Every other phase, and a missing status, keeps the wait open until the install window expires. There is no per-component timeout, so a download longer than that window is retried as an upgrade until it finishes, and the retry strategy never uninstalls the volumes. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM |
||
|
Marian Koreniuk
|
93c89c8bf6 |
feat(kubernetes): add the Proxmox CSI driver, with its controller in this cluster
Tenant clusters on the proxmox substrate get persistent volumes backed by the
hypervisor's storage. The driver is sergelogvinov/proxmox-csi-plugin, vendored
as a chart that reproduces byte-for-byte from `helm pull`.
It is deployed in halves, and which half goes where is the whole design. The
controller authenticates to Proxmox, so it runs HERE, in the tenant's namespace
of the management cluster, and drives the tenant over its admin kubeconfig —
the same shape templates/csi/deploy.yaml already uses for kubevirt-csi. The
tenant gets the node plugin, the CSIDriver object and the storage classes; its
DaemonSet mounts no credentials.
Putting the controller in the tenant would hand the token to the tenant. The
resource definition lists the admin kubeconfig in secrets.include, so the
tenant is cluster-admin of its own cluster and can read any Secret there, and
the chart writes the driver's config.yaml — token included — wherever it is
installed. The privileges the driver documents are granted on `/`, so that
token reaches every VM and every volume on the Proxmox cluster, not just this
tenant's.
Three consequences of the split are handled here rather than discovered later:
- The vendored chart has no switch for its controller, so the tenant release
sets replicaCount: 0. The Deployment object still renders; no Pod does.
- Its Secret template is gated on a non-empty config.clusters, so an empty
list is what keeps the credential out.
- CSIStorageCapacity is published by the provisioner sidecar, which now runs
here and owns those objects through a Pod that does not exist in the
tenant. Left on, CSIDriver.storageCapacity=true and the scheduler refuses
every Pod with "node(s) did not have enough free storage". The chart
hardcodes that field, so a kustomize postRenderer patches it rather than
forking the chart.
The controller follows proxmox.csi.enabled, like the tenant release: with the
driver switched off there is nothing for it to serve and no token to hold.
Volumes are owned per tenant. The driver names them vm-<controllerVmID>-pvc-…
and defaults the id to 9999; on shared storage that made every tenant's volumes
indistinguishable to an operator and to a VM-level ACL. Each cluster derives its
own from its release name.
What this does not do is make one tenant unable to reach another's volumes: a
tenant can still create a static PV naming a foreign volumeHandle. That needs a
per-tenant Proxmox scope — a dedicated storage and a VM ACL per tenant — which
is a deployment requirement rather than a chart change.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
Aleksei Sviridkin
|
fd8ccb2fcb |
feat(platform): keep VMI-only hotplugged volumes across declarative hotplug
Dropping the HotplugVolumes gate lets virt-controller rebuild each running VMI's hotplug volumes from VM.spec, and it removes every hotpluggable VMI volume the VM spec does not list. kubevirt-csi attached tenant volumes at the VMI level until mid-2025 and live migration carries such a VMI across KubeVirt updates, so a tenant node that was never recreated can still hold a mounted disk only its VMI knows about; the first reconcile after the upgrade would unplug it. Migration 59 writes each such volume and its disk into the VM template as they are on the VMI, before the upgrade re-points cozystack-packages and so before the KubeVirt CR and the kubevirt-csi flag change. Hotplug volume changes are a valid live update, so no VM needs a restart. An ephemeral hotplug a user made at the VMI level, such as virtctl addvolume without --persist, has the same shape, because virt-api gives every VMI-level addvolume that shape. It is copied too and becomes permanent: leaving it out would not keep it ephemeral, since the first reconcile would unplug it. Only Running VMIs are considered, matching virt-controller, which reconciles no other; a stopped VM's next start builds its VMI from the VM spec, so its VMI-only volumes are not resurrected. Only the variants with migrations enabled run it. The default variant renders no migration hook, so a cluster there that installed KubeVirt by hand has to check for such volumes itself before upgrading. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
IvanHunters
|
30782d003c |
fix(platform): attach cozy-lib to the bucket-system package source
system/bucket now calls cozy-lib.ns-host-required, but in a cluster the chart is built from the ExternalArtifact per its PackageSource, not from the charts/cozy-lib symlink used by helm and helm-unittest. The bucket-system component did not list cozy-lib, so the built chart lacked the helper and every cozy-bucket install (including the platform backups bucket) failed to render. Attach the library to the component, as the apps/bucket component already does. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |