mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
backup-clickhouse-e2e
3656
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Andrey Kolkov
|
ade3bba33d |
feat(platform): add e2e tests for backup altinity strategy for clickhouse
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
70284d57f5 |
feat(platform): implement backup strategy for clickhouse application
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrei Kvapil
|
99e7070e4e |
fix(seaweedfs): wait for ingress-nginx-system before creating Ingress (#2406)
## What this PR does Stacks on top of #2258. Adds `dependsOn: ingress-nginx-system` to the inner `seaweedfs-system` HelmRelease so flux blocks its install until the ingress chart is Ready. ## Why In #2258's E2E run (and the `cozyreport-2258` bundle from this morning), `seaweedfs-system` failed to install with: ``` Helm install failed for release tenant-root/seaweedfs-system: 1 error occurred: * Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://root-ingress-controller-admission.tenant-root.svc:443/networking/v1/ingresses?timeout=10s": dial tcp 10.96.33.42:443: connect: operation not permitted ``` ### Root cause class The outer tenant HelmReleases (`apps/tenant/templates/*.yaml`) render one inner HR per subsystem and let flux reconcile them. Both `seaweedfs` and `ingress` outer HRs get applied in parallel, and each of them spawns its own inner HR — which flux then installs in parallel too. When `seaweedfs-system` gets to the `creating 39 resource(s)` step, it tries to create the tenant Ingress. The apiserver calls the ingress-nginx `ValidatingWebhookConfiguration` (which uses `failurePolicy: Fail` by default). If the nginx controller pod/Service/endpoint state hasn't finished reconciling on the CNI yet, the connect to the ClusterIP fails with `EPERM`, the webhook errors, Helm aborts, and flux rolls back. E2E recovers via the `flux reconcile --force` retry loop in `hack/e2e-install-cozystack.bats`, but it's flaky and adds minutes. ### Fix Introduce an explicit ordering dependency at the inner HR layer: ```yaml {{- if $ingress }} dependsOn: - name: ingress-nginx-system {{- end }} ``` Flux waits for `ingress-nginx-system` to reach `Ready=true` before installing seaweedfs — which, under Flux's wait semantics, means the ingress-nginx Deployment is Available and its webhook endpoint is routable. The race disappears. Guarded by `{{- if $ingress }}` so this only kicks in for tenants that actually have ingress enabled. ## Test plan - [ ] Rerun E2E on #2258 — `seaweedfs-system` should install on the first try without a `Retry install` iteration. - [ ] Verify rendered template still matches expectations when `$ingress` is unset (no `dependsOn` block). ### Release note ```release-note [seaweedfs] Make tenant seaweedfs wait for ingress-nginx to be Ready before creating its Ingress resource, avoiding a webhook race that intermittently failed install on fresh clusters. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved deployment sequencing so the system now waits for the Ingress controller to be ready before creating or updating resources that include Ingress. This prevents premature deployments and related failures, reduces rollout errors, and provides more reliable, predictable deployments for applications relying on Ingress resources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
45e21a330f |
feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561)
## What this PR does Adds [`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0) — a Go reimplementation of [`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy) — and wires it into cozystack as a *conditional default*: a single `publishing.proxyProtocol: true` switch turns PROXY-protocol on at the host ingress-nginx **and** auto-deploys ouroboros to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arriving at ingress-nginx without the PROXY header it now requires). Default stays `false` — clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. The chart is platform-agnostic at v0.7.0: the controller and proxy binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf` inside the pod and compose the rewrite target FQDN and the backend host at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on `cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`, federations on `k8s.example.com`, custom-domain RKE2/k3s — all work without operator override. Verified end-to-end on a `cozy.local` cluster: controller logs `cluster-domain=cozy.local` at startup, chart emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked `--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy ClusterIP. Operators that prefer a chart-baked FQDN can still set `controller.clusterDomain` explicitly — the explicit-override path is preserved for non-Service backends and for split-horizon DNS topologies. The chart runs in `coredns` mode for the host (the host CoreDNS is Talos-managed and does not carry an `import` directive, so the alternative would silently no-op there) and in `coredns-import` mode for tenant deployments (tenants run cozystack-coredns, which carries the import directive and ships an empty `kube-system/coredns-custom` ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the `ouroboros.override` data key without ever touching the chart-rendered Corefile, so it does not race the chart that re-templates `data.Corefile` on every Flux reconcile). ### Layout * `packages/system/ouroboros/` — vendored chart from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`: `controller.mode=coredns` (host scenario), `proxy.target.namespace=cozy-ingress-nginx` (override the chart-default `ingress-nginx` namespace; chart-default `serviceName=ingress-nginx-controller` matches as-is), Gateway-API watching left at upstream's default of `false`. `proxy.target.host` is left empty so the proxy composes the backend host at runtime. Image is digest-pinned at `0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c` and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is intentionally not mirrored under `ghcr.io/cozystack/*`. The chart manifest itself is also digest-pinned via `OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e` in the package Makefile, with a Chart.yaml-version vs. Makefile-version assert that aborts the update target on lockstep drift. Helm-unittest in `tests/` covers controller args, RBAC narrowing per mode, Gateway-API gating, runtime-composition flag emission in both `coredns` and `coredns-import` modes, image pin drift. * `packages/system/coredns/templates/coredns-custom.yaml` + `values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed `import` prerequisite, plus the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts` mount at `/etc/coredns/custom/`. Same package serves host and tenant CoreDNS, so this covers both layers without duplication. Helm-unittest pins the rendered Corefile via a snapshot so future `make update` does not silently drop the import directive. The empty rendered ConfigMap deliberately omits `data:` — Helm three-way merge has no template-side keys to compare against, so apiserver-side runtime writes (the `ouroboros.override` data key) survive every chart upgrade. The `notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the invariant. * `packages/core/platform/sources/ouroboros.yaml` — new `cozystack.ouroboros` PackageSource for the host install (`dependsOn: cozystack.networking, cozystack.ingress-application`). * `packages/core/platform/sources/kubernetes-application.yaml` — `kubernetes-ouroboros` component registered against `system/ouroboros` so the tenant HelmRelease can reference it. * `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` — tenant-side HelmRelease deployed via the tenant admin-kubeconfig, mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues` helper flips `mode` back to `coredns-import` for tenants and mirrors per-tenant `addons.gatewayAPI.enabled` into the controller (forcing `--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the controller on `WaitForCacheSync`). `controller.clusterDomain` is intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from `/etc/resolv.conf` and composes the rewrite target at runtime, so the same wrapper renders correctly on `cluster.local`, `cozy.local`, or any custom tenant cluster-domain. Operators on a non-default cluster-domain can still override via `valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a conditional `gateway-api-crds` entry on the same gate. If `addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled` is false, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool `true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML `yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via `toString`-then-`lower`-then-string-compare; direct bool `eq` would error on type mismatch when an operator passes the value as a string from `--set`, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. Tenant disable is single-step: flipping `addons.ouroboros.enabled: false` deletes the rendered HelmRelease, helm-controller runs `helm uninstall` on the tenant, and the chart's pre-delete cleanup hook (`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the `ouroboros.override` key on the tenant's `kube-system/coredns-custom` ConfigMap automatically. No render-time guard is wired on the tenant side — gating on a `lookup` for the leftover HelmRelease would deadlock the routine disable (the parent render runs before helm-controller applies the missing-child diff, so the lookup would always find the leftover at the moment of the flag flip). If the chart's hook fails to land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job timeout), operators recover manually via the documented tenant cleanup recipe. * `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef with `enabled` + `valuesOverride`. `make generate` re-emitted README, JSON schema, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. * `packages/core/platform/values.yaml` — new `publishing.proxyProtocol` flag with extensive comment covering the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on, otherwise external traffic breaks), the host-vs-tenant fan-out (the flag does not cascade into tenants), the Talos re-render flap window, and the asymmetric disable path. The host disable path: flipping back to `false` reverts ingress-nginx PROXY-protocol config (flip the LB off first), but the host ouroboros itself stays installed because its Package CR carries `helm.sh/resource-policy: keep` — an explicit `kubectl delete package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages` entry) actually triggers helm uninstall, at which point the chart's pre-delete cleanup hook fires and patches the host Corefile automatically. The values comment ships a manual `sed`-based cleanup recipe as a fallback for the rare case where the chart's hook fails to land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off a `lookup` against the leftover Package CR to catch operators who flip the flag back without confirming the cleanup landed. * `packages/core/platform/templates/bundles/system.yaml` — flips between the default-form and explicit-form `cozystack.ingress-application` Package emission so the host ingress-nginx gets `use-proxy-protocol` + `real-ip-header=proxy_protocol` injected only when the flag is true (`use-forwarded-headers` and `compute-full-forwarded-for` are deliberately *not* on the list — they would let any upstream proxy spoof `X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`. Host Gateway-API watching is left at the upstream chart's default of `false` because cozystack does not install Gateway-API CRDs cluster-wide on the host. * `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in tenant, controller pod running, and after creating a TLS Ingress the rewrite snippet shows up in the import ConfigMap; an in-tenant `dig` from a netshoot pod also confirms CoreDNS actually serves the rewrite — pod-readiness alone passes vacuously since the upstream chart ships no readiness probes). ### Why not KEP-1860 `ipMode=Proxy` Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB frontend entirely when `ipMode != VIP`, breaking Service ingress for L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters fronted by an external CCM-managed proxy LB. ouroboros is the correct fix for L2/BGP-announce topologies that cozystack ships by default. ### Coverage Covered by helm-unittest (host chart, coredns wrapper, tenant addon — including the `kubernetesProvider` mock that exercises the lookup-driven `proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil CI/dry-run path, the gateway-api truthy-spelling acceptance across `true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that `--proxy-service-name` + `--proxy-service-namespace` are emitted instead of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import` modes, the assertion that `--target-service-name` + `--target-service-namespace` are emitted for the proxy backend, and the `_namespace.etcd`-empty render gate) and a tenant e2e bats test that creates a real Ingress, asserts the rewrite snippet lands in the import ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry loop tolerating CoreDNS reload latency). The host-level `publishing.proxyProtocol: true` path is not e2e-tested in this PR — it requires a PROXY-protocol-aware upstream LB, which the cozystack e2e harness does not stand up. That path is exercised manually and by docs review. End-to-end runtime auto-detect was verified manually on a `cozy.local` cluster (dev17): chart deployed at v0.7.0, controller logged `cluster-domain=cozy.local`, the rewrite block landed in `kube-system/coredns` Corefile pointing at `test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig <ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first attempt, and the chart's pre-delete hook restored the Corefile cleanly on `helm uninstall`. ### Supply-chain note Image and chart are pulled from a contributor's personal namespace (`ghcr.io/lexfrei/...`). Existing precedent (`packages/system/etcd-operator/Makefile` pulls from `ghcr.io/aenix-io/...`) is similar but `aenix-io` is the cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer call. Pairs with cozystack/website#527 (operator docs). ### Release note ```release-note feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with configurable enablement and tenant overrides. * Added a packaged Ouroboros chart providing proxy, controller, CoreDNS integration, external‑DNS and etc‑hosts modes. * Introduced PROXY‑protocol gating with an explicit acknowledgement flag. * **Documentation** * Expanded addon reference and platform docs with detailed Ouroboros/PROXY‑protocol configuration and defaults. * **Tests** * Added extensive unit and e2e tests covering rendering, RBAC, uninstall cleanup, and DNS behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
8da1e7a40a |
feat(platform): implement backup strategy for postgres (#2552)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note Implemented CNPG-strategy controller for postgres backups with supporting in-place and to-copy restores. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * CNPG-based backup strategy for PostgreSQL with S3-backed object store and controller support * Option to reference existing Kubernetes Secret for S3 credentials (configurable key names) * Controller supports scheduled/ad-hoc backups and both in-place and to-copy restores; safer credential handling (no inline keys) * **Examples** * Complete manifests: bucket, credentials, CNPG strategy, BackupClass, Plan, BackupJob, RestoreJob, Postgres source/target * **Security/RBAC** * Controller RBAC expanded for CNPG resources; tests enforce no cluster-wide Secret read * **Documentation** * README, chart values/schema and UI schema updated to document secret-reference workflow and restore guidance * **Tests** * New end-to-end and unit tests covering CNPG flows, restore semantics, cleanup behavior and RBAC constraints <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
653519a472 |
fix(coredns): sanitize Chart.Version build metadata in coredns-custom label
The cozystack build pipeline tags chart versions as `0.0.0+<git-sha>`. The new coredns-custom ConfigMap interpolated `.Chart.Version` directly into the `helm.sh/chart` label, producing `cozy-coredns-0.0.0+<sha>` which the apiserver rejects: label values follow RFC 1123 and `+` is not a legal character. The chart never installed, kube-system/coredns inside tenant clusters never came up, every dependent HelmRelease stalled, and the e2e suite hit the 2h job timeout without reaching the bulk of the bats tests. Apply the standard `replace "+" "_" | trunc 63 | trimSuffix "-"` sanitization, matching the pattern the vendored upstream coredns chart already uses in its own `helm.sh/chart` helper a few directories down. Add a helm-unittest case that overrides the chart version to a build- metadata form (`0.0.0+abc1234567`) and asserts the rendered label both matches the sanitized literal and conforms to the RFC 1123 label-value regex. Without the version override the bug is invisible to the test suite — Chart.yaml ships the bare `0.0.0` placeholder and helm-unittest defaults to that, so any future template that interpolates `.Chart.Version` into a label value cannot regress past CI silently. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
026581e0e9 |
fix(seaweedfs): only depend on ingress-nginx-system when tenant owns its ingress
Address review blocker on #2406: gate the dependsOn on `eq $ingress .Release.Namespace` so sub-tenants inheriting ingress from a parent namespace don't deadlock waiting for an HR that doesn't exist locally. Also add explicit `namespace:` to match repo convention. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
43c76fd8f3 |
fix(seaweedfs): wait for ingress-nginx-system before creating Ingress
Without this, the outer tenant HelmReleases apply both seaweedfs and
ingress in parallel, which in turn creates their inner HelmReleases
(seaweedfs-system and ingress-nginx-system) in parallel too. When
seaweedfs-system tries to create its Ingress before the nginx admission
webhook's Service/endpoint state has finished reconciling on the CNI,
the apiserver's webhook call fails with:
failed calling webhook "validate.nginx.ingress.kubernetes.io":
dial tcp ...:443: connect: operation not permitted
and flux rolls back the install. E2E recovers via the "Retry install"
loop but it's flaky and adds minutes to every run.
Adding dependsOn: ingress-nginx-system on the inner seaweedfs HR makes
flux block the install until the ingress chart is Ready, which means
the controller Deployment is Available and its webhook endpoint is
routable before the first Ingress is created.
Guarded by {{- if \$ingress }} so this only kicks in for tenants that
have ingress enabled (i.e. tenant-root in the default topology).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
myasnikovdaniil
|
a7a09ff5b2 |
docs(release): add changelog for v1.3.3 (#2594)
This PR adds the changelog for release `v1.3.3`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.3.3.md`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added v1.3.3 release notes documenting PostgreSQL backup recovery
improvements, Kafka and MongoDB managed application fixes, resource
preset updates, and a new vGPU setup guide.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Myasnikov Daniil
|
bc19eae311 |
docs: add changelog for v1.3.3
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
c22d6661ad |
fix(kafka): create LoadBalancer service when external flag is enabled (#2578)
## What this PR does When `external: true` is set in the Kafka chart, Strimzi creates a LoadBalancer service named `<release>-kafka-external-bootstrap` for external client access. However, two issues prevented this from working correctly: 1. **Unconditional listener definition**: the `external` listener was always present in the Kafka CR (as `type: internal` when disabled), instead of being omitted entirely when `external: false`. 2. **Missing dashboard RBAC**: the dashboard role only granted access to `<release>-kafka-bootstrap` (internal ClusterIP). The Strimzi-managed external bootstrap service was not included, making it invisible to tenant users even when it was created. **Changes:** - Make the external listener conditional — added only when `external: true` - Add `<release>-kafka-external-bootstrap` to dashboard RBAC rules when `external: true` - Add helm-unittest tests covering listener and RBAC behavior for both `external: true` and `external: false` ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(kafka): external listener is now correctly created as a LoadBalancer and exposed in the tenant dashboard when `external: true` is set ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * External Kafka listener and external bootstrap service in dashboard resource mapping are now conditionally enabled based on configuration settings. * **Tests** * Added comprehensive test suites validating Kafka configuration rendering and dashboard resource mapping across internal and external deployment variations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
ed5ae16e36 |
feat(platform): implement backup strategy for postgres application
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Aleksei Sviridkin
|
a9c1298b42 |
[gpu-operator] Update to v26.3.1 and add experimental vGPU variant (#2323)
## What this PR does Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an **experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and newer GPUs. Changes: - Update vendored upstream chart to v26.3.1 (CDI enabled by default, component version bumps). - Pin chart version in Makefile for reproducibility. - Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload: vm-vgpu` so the chart activates the vGPU Manager DaemonSet without per-node labelling. - Add `vgpu` variant to PackageSource alongside the existing `default` (passthrough). - Rename `values-talos.yaml` → `values-passthrough.yaml` and pin `defaultWorkload: vm-passthrough` for symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically. - Refresh `hack/dcgm-default-counters.csv` to match the new dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED` and other fields introduced upstream. `hack/check-gpu-recording-rules.bats` passes against the new fixture. - Add `hack/check-gpu-operator-variants.bats` that renders both variants via `helm template` and asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them. - Add internal documentation for the vGPU bring-up. The `vgpu` variant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed. ### `vgpu` variant is experimental NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (`echo <id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after every node reboot. Without it, `permittedHostDevices.pciHostDevices` reports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR. ### KubeVirt requirement KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which is `main`-only — released tags `v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned. ### Behavior changes inherited from upstream chart v25 → v26 The upstream chart flipped two defaults that are pinned back to `false` here, so `default` (passthrough) deployments do not silently change footprint on upgrade: - `ccManager.enabled` was flipped to `true` upstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade. - `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet would render even in passthrough mode. Both stay opt-in via Package CR overrides. The new bats test guards the pin. Other inherited changes that are user-visible and worth pinning down before upgrade: - `driver.licensingConfig.configMapName` is deprecated upstream in favour of `driver.licensingConfig.secretName`. The deprecation marker lives in the CRD's `description` field (visible via `kubectl explain`) — there is no helm-render-time warning, the chart template silently prefers `secretName` and falls back to `configMapName`. Operators with existing licensing overrides should migrate the key on the next chart upgrade. - `driver.version` upstream default jumped from `570.x` to `580.x`. Invisible on the `default` variant (which sets `driver.enabled: false`), user-visible for anyone running the upstream Ubuntu driver path. - `vfioManager.image` switched from `nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every `default` (passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters. - `dcgmExporter.image` switched to the `-distroless` flavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used to `kubectl exec` into the exporter for ad-hoc debugging will need to switch to `kubectl debug` or container ephemeral images. - `CDI` (Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is. - `node-feature-discovery` subchart source switched from the GitHub Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped installations need to mirror the new OCI URL. ### Talos Talos is **not recommended** for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (`default` variant) on Talos is unaffected. ### Release note ```release-note [gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced vGPU support variant with end-to-end configuration guide, validation procedures, and troubleshooting documentation * **Updates** * Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3 * Improved component health checks and added PodDisruptionBudget availability controls <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
6cc9013085 |
docs: add changelog for v1.3.2 (#2550)
This PR adds the changelog for release `v1.3.2`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.3.2.md`.
untagged-34c5e929a0cb9154bef0
|
||
|
myasnikovdaniil
|
61de813613 |
feat(e2e): parallel post-install prep (#2557)
## What this PR does ### Parallel post-install prep LINSTOR / StorageClass / MetalLB setup, previously sequential `@test` cases after the 15 m platform HR wait, now run as a background prep that completes during the platform reconcile. None of them gates platform HR reconcile, so they can move out of the critical path. New helper `hack/e2e-post-install-prep.sh` polls for its prerequisites and is awaited at the end of the `Install Cozystack` test so failures still surface deterministically. ### T2-race fix Wait on the LINSTOR HR being `Ready` before polling for `deploy/linstor-controller`. The old 5-min hard timeout fired during cold installs where the LINSTOR HR's dependency chain (cert-manager → piraeus-operator-crds → piraeus-operator → linstor) takes longer than 5 min to resolve. ### E2E uses helmReleaseInterval=30s E2E install passes `--set cozystackOperator.helmReleaseInterval=30s` to eliminate the 8–10 min Flux dead zone where dependency-blocked HRs are requeued only every 5 min. Operator-side flag and chart value landed in #2509; production defaults are unchanged. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** - Improved end-to-end test initialization to run concurrent post-install preparation, wait for cluster release reconciliation, verify component readiness, and validate automated storage configuration. * **Chores** - Added background post-install setup and logging to increase test reliability and speed up cluster initialization and teardown diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
5096e3bf9c |
fix(ci): stop backport workflow from cascading on its own PRs (#2584)
## What this PR does Fix the cascade where merging a backport PR (e.g. #2579) spawns another backport (e.g. #2581 `[Backport release-1.2] [Backport release-1.3] …`). Two compounding issues: 1. `pr-labeler.yaml` auto-added the `backport` label to any PR whose title started with `[Backport release-X.Y]` — exactly the prefix `korthout/backport-action` puts on every backport PR. Introduced in `c8ed1c65` (CNCF/k8s label conventions). Fixed by dropping that `toAdd.add('backport')` line. 2. `dosubot[bot]` independently re-applies `backport-previous` to backport PRs based on similarity to the parent PR; it cannot be controlled from this repo. Fixed by gating both jobs in `backport.yaml` on `pull_request.base.ref == 'main'`, so backport PRs (which target `release-X.Y`) cannot trigger the workflow regardless of labels. Verified pre-fix behaviour by inspecting the label-event timeline of #2580 (dosubot adds `backport-previous` 47s after PR open) and the diff of `c8ed1c65` adding `pr-labeler.yaml`. ### Release note ```release-note fix(ci): backport automation no longer creates duplicate backport PRs when a backport PR is merged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Enhanced backport automation workflow with additional safety checks to ensure operations only process pull requests targeting the main branch. * Simplified pull request labeling workflow to streamline label assignment logic. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
3e9e393b7f |
fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts (#2555)
## What this PR does Three fixes for `hack/e2e-apps/vminstance.bats`, surfaced once the 3x retry was dropped from `Run E2E tests`: 1. **Disk delete-recreate race.** The teardown used `kubectl delete vmdisk … || true` and immediately created a new disk; sometimes the new disk reconciled before the old finalizer completed, leaving a duplicate PVC and the new VM stuck. Drop the `|| true`, block until the old disk is gone, then create. 2. **VM IP wait timeout 20s → 120s.** Under nested-virt + concurrent runner load, a fresh VM sometimes takes 60–90 s to acquire its IP via DHCP. The 20 s bound was tripping deterministically. 3. **VM ready timeout 20s → 60s.** Same root cause; the readiness probe needs the IP to be reachable, so the lower bound has to grow alongside the IP wait. Existence backstop on the `vm-instance-$name` HR is added here too (it touches the same hunk as the timeout fix; cleaner to land them together). Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved end-to-end test reliability for VM instance and disk flows. * Replaced short sleeps with explicit waits for upstream and downstream deployment artifacts. * Extended polling windows and timeouts for IP assignment and readiness checks. * Strengthened cleanup to reliably remove prior VM and disk resources before and after tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
cozystack-ci[bot]
|
9d7941b9ec |
docs: add changelog for v1.3.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
32e75ab183 |
Merge branch 'main' into daniil/split-vminstance
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
myasnikovdaniil
|
7585b0c42c |
test(e2e): replace fixed timeouts with event-driven backstops (#2554)
## What this PR does Replaces fragile `sleep N` / `timeout N until …; do sleep …; done` patterns in `hack/*.bats` and `hack/*.sh` with `kubectl wait --for=condition=…` plus a small existence backstop: ```sh until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done kubectl wait <resource> --for=condition=Ready --timeout=… ``` ### Why the existence backstop matters `kubectl wait` against a not-yet-created resource sits silently for the full timeout and then fails with `error: timed out waiting for the condition` — useless for diagnosis. With the backstop, the test fails fast with a clear "resource never appeared" error. Existence backstop added at 14 sites in this sweep across: - `redis`, `kafka`, `openbao`, `harbor`, `mongodb`, `qdrant`, `clickhouse`, `mariadb`, `postgres`, `bucket`, `external-dns`, `foundationdb` bats - `hack/e2e-install-cozystack.bats` - `hack/e2e-apps/run-kubernetes.sh` - `hack/e2e-test-openapi.bats` (port-readiness wait replacing fixed `sleep 5` after `kubectl proxy`) The vm-disk hunk in `hack/e2e-apps/vminstance.bats` is included here; the vm-instance hunks land separately via the `daniil/split-vminstance` PR. ### Genuine sleeps annotated The 3 remaining genuine sleeps (LINSTOR node membership propagation, MetalLB end-to-end HTTP path, HR-count emission heuristic) are annotated with `# TODO` + rationale so a future reader knows they are intentional. Same green-path semantics; failures now surface in seconds instead of after a 5-minute timeout. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved e2e stability by replacing fixed sleeps with timeout-based polling across many integration tests, waiting for operator-created resources to exist before readiness checks to reduce race conditions and flakes. * **Documentation** * Added clarifying notes for LoadBalancer retry behavior and improved local proxy readiness checks in e2e scripts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
af317f16a4 |
fix(ci): stop backport workflow from cascading on its own PRs
Two compounding issues caused merging a backport PR to spawn another
backport (e.g. #2581 [Backport release-1.2] [Backport release-1.3] ...
from #2579):
1. pr-labeler auto-added the `backport` label to any PR whose title
started with `[Backport release-X.Y]` (regression introduced in
|
||
|
myasnikovdaniil
|
8e78536cc3 |
fix(cert-manager): bump chart to v1.20.2 (#2562)
## What this PR does Bumps the vendored cert-manager chart from v1.19.3 to v1.20.2. Pure upstream bump via `make update` — no cozystack-side template changes. ### Why [Commit `2b6e20cc`](https://github.com/cozystack/cozystack/commit/2b6e20cc) migrated the platform's ACME HTTP-01 setup to the modern `ingressClassName` API on both the ClusterIssuer's solver and the per-tenant override annotation `acme.cert-manager.io/http01-ingress-ingressclassname`. The override annotation, however, was only added to upstream cert-manager in v1.20.0 ([cert-manager#8244]) — v1.19.x silently ignores it. Result: cert-manager always falls back to the ClusterIssuer's solver class (`tenant-root`), so HTTP-01 solver Ingresses for non-`tenant-root` tenants land on the wrong controller and Let's Encrypt validation fails for tenant-scoped Ingresses (grafana, harbor, alerta, etc. inside child tenants). Targeting v1.20.2 (not .0): - [cert-manager#8655] (in v1.20.1) — restores `clusterissuers/issuers/finalizers` RBAC dropped in v1.20.0. - [cert-manager#8665] (in v1.20.2) — webhook helm-template fix. ### Verified live on dev10 | Stage | `Challenge.spec.solver.http01.ingress.ingressClassName` | |---|---| | Pre-bump (v1.19.3) | `"tenant-root"` (bug) | | Post-bump (v1.20.2) | `"<tenant>"` (fixed) | Full propagation chain confirmed end-to-end: source Ingress annotation -> ingress-shim sets `http01-override-ingress-ingressclassname` on the Certificate -> acmeorders sets `spec.solver.http01.ingress.ingressClassName` on the Challenge -> http01 solver creates the solver Ingress on the tenant's class. ### Upgrade considerations - **Container UID/GID change**: 1000/0 -> 65532/65532 across controller, webhook, cainjector. PSP / SecurityContext rules pinning UID 1000 will need updating. - **Helm chart image structure** refactored from `image.repository` to `imageRegistry` + `imageNamespace` + `image.name`. Cozystack's wrapper `values.yaml` is empty, so rendered image references are unchanged (`quay.io/jetstack/cert-manager-{controller,webhook,cainjector,acmesolver}:v1.20.2`). - **NetworkPolicy templates restructured**: legacy `networkpolicy-egress.yaml` removed; per-component `networkpolicy-cainjector.yaml` and `networkpolicy-cert-manager.yaml` added. All gated on `*.networkPolicy.enabled: false` (default). No behavioral change. - **CRDs**: `selectableFields` added (Kubernetes >=1.30) and Azure DNS01 `zoneType` enum. Both pure additions. - **RBAC**: gains `clusterissuers/issuers/finalizers` and `gateway.networking.k8s.io.listenersets[/finalizers]`. - `DefaultPrivateKeyRotationPolicyAlways` feature gate moves Beta -> GA, can no longer be disabled. Already default since v1.18; no behavior change. ### Release note ```release-note fix(cert-manager): bump to v1.20.2 - fixes HTTP-01 solver IngressClass propagation for tenant-scoped Ingresses (the `acme.cert-manager.io/http01-ingress-ingressclassname` annotation was a no-op in prior cert-manager versions). Default container UID/GID changes from 1000/0 to 65532/65532 - review your PSP/SecurityContext rules. ``` [cert-manager#8244]: https://github.com/cert-manager/cert-manager/pull/8244 [cert-manager#8655]: https://github.com/cert-manager/cert-manager/pull/8655 [cert-manager#8665]: https://github.com/cert-manager/cert-manager/pull/8665 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added network policy support for controller, webhook, and CA injector * Added Azure DNS zone type selection (public/private) * Added webhook client verification configuration options * Extended pod disruption budget configuration with unhealthy pod eviction policy * **Updates** * Upgraded cert-manager to v1.20.2 * Enhanced global image registry and namespace configuration * Expanded issuer reference field selectors * Added support for extra containers in controller pod * Updated issuer terminology and configuration * **Documentation** * Updated Helm chart documentation with new configuration options * Updated post-installation guidance <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
418e3da527 |
fix(kafka): create LoadBalancer service when external flag is enabled
When `external: true` was set, the Kafka chart relied on Strimzi to create a LoadBalancer service via listener configuration, but the listener was unconditionally defined (as `type: internal` when disabled). Additionally, the dashboard RBAC did not include the Strimzi-managed external bootstrap service, making it invisible to tenant users. - Make the external listener conditional: only add it to the Kafka CR when `external: true` is set - Add `<release>-kafka-external-bootstrap` to dashboard RBAC rules when `external: true`, so the LB service is accessible from the tenant dashboard - Add helm-unittest tests for both kafka listeners and dashboard RBAC covering the external/internal paths Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniil
|
d8da80bdd2 |
fix(e2e): propagate failures from parallel LINSTOR pool jobs
Address review feedback on hack/e2e-post-install-prep.sh:31: POSIX `wait` without arguments returns 0 even if a backgrounded `kubectl exec` failed, which would mask partial LINSTOR prep failures and let downstream tests run on a half-provisioned cluster. Collect each background PID and `wait "$pid"` per job so `set -e` exits the script on the first failed pool creation. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
209fa08737 |
test(e2e): decouple vminstance test from vmdisk test
"Create a VM Instance" referenced the VMDisk created by "Create a VM Disk" via the shared name "test". This implicit ordering meant any defensive pre-clean of vmdisks in test 2 would destroy the disk test 1 just provisioned, and the vm-instance chart's lookup-or-fail on the DataVolume would then surface as "Specified disk not exists in cluster: test". Make each test self-contained: - "Create a VM Disk" cleans up its own VMDisk at the end. - "Create a VM Instance" creates its own VMDisk inline (renamed to "vmi-test" to avoid colliding with any in-flight finalizer drain on test 1's resources) and waits HR -> DV -> PVC before applying the VMInstance. Replaces the pre-delete approach from the prior followup commit, which broke the data dependency. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
585ea7e089 |
test(e2e): smoke-test the ouroboros tenant addon
Auto-discovered hack/e2e-apps/ouroboros.bats spins up a tenant
Kubernetes CR with addons.ingressNginx + addons.ouroboros enabled
and asserts the cozystack-side wiring delivers a working hairpin
fix end-to-end:
* the cozy-ouroboros HelmRelease in the tenant's host namespace
reaches Ready (chartRef + admin-kubeconfig wiring is correct);
* the cozystack-coredns wrapper rendered the kube-system/coredns-custom
ConfigMap inside the tenant (the import-glob prerequisite is in
place);
* the controller pod is running in cozy-ouroboros inside the tenant;
* after creating an Ingress with a TLS hostname, the tenant's
coredns-custom ConfigMap eventually carries a `rewrite name <host>`
line under the ouroboros.override key — proves the controller is
reconciling, not merely Pod-Ready (the upstream chart ships no
readiness probes, so a controller stuck on WaitForCacheSync would
pass a pod-readiness check vacuously; the rewrite assertion catches
every cozystack-specific failure mode the unit tests cannot — wrong
namespace, missing CRDs, narrowed RBAC, gateway-api gate misfires).
End-to-end PROXY-protocol header roundtrip (the actual hairpin fix
in action) is covered by the upstream chart's own e2e on every push;
this file is the cozystack-side integration test that the cozystack
glue (chart vendoring, kubernetes-application registration, tenant
HelmRelease wiring, gateway-api gating, coredns-custom prerequisite)
holds together.
cozystack hack/cozytest.sh is plain shell, not real bats, so kubectl
output is captured directly without the `run` helper.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
7f01c7eb32 |
feat(platform): single switch for host PROXY-protocol + ouroboros
Adds publishing.proxyProtocol on the platform values: a single flag that both turns PROXY-protocol on at the host ingress-nginx (via the ingress-application Package CR's component values dict) and emits the new cozystack.ouroboros Package CR for the host install. The two pieces are coupled by design — turning PROXY-protocol on at ingress-nginx without ouroboros breaks intra-cluster traffic to the cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits the LB IP, the connection arrives at ingress-nginx without the PROXY header it now requires, and ingress-nginx closes the connection). The new sources/ouroboros.yaml PackageSource declares the host chart with dependsOn: cozystack.networking + cozystack.ingress-application so install ordering is deterministic. The bundle template emits the host ouroboros Package via the standard helper — the parent chart's Gateway-API watching default of false stays in place, because cozystack does not install Gateway-API CRDs cluster-wide on the host (system/ gateway-api-crds is referenced only from the kubernetes-application PackageSource, tenant scope). Forcing --gateway-api on a host without the CRDs would CrashLoop the controller on WaitForCacheSync against missing GVRs. The flag default is false. Clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. Disabling: flipping publishing.proxyProtocol back to false stops emitting the cozystack.ouroboros Package CR, but every Package CR carries helm.sh/resource-policy: keep — Helm leaves the existing Package on the cluster so ouroboros stays installed and the live Corefile rewrite block keeps pointing at the still-running ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To actually uninstall ouroboros the operator removes the Package CR explicitly (or adds the package name to bundles.disabledPackages) and runs the documented Corefile cleanup recipe in the same window — otherwise stale rewrites point at a deleted Service and in-cluster DNS for hairpinned hostnames returns connection-refused. The values comment ships the cleanup recipe (a JSON merge-patch on data.Corefile that strips the BEGIN/END block while preserving labels, annotations, and other data keys including any Talos-managed metadata) and documents the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on), the host-vs-tenant fan-out (the flag does not cascade into tenant clusters; tenants opt in per-Kubernetes-CR via addons.ouroboros.enabled), and the Talos re-render flap window (machine-config pushes wipe the BEGIN/END markers; the controller re-applies on next reconcile, ≤10 minutes by default). Also drops the stale "distro-full" entry from the bundles.system.variant options comment in core/platform/values.yaml — the variant was removed in v1.0.1 and was never restored. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7ab940f309 |
feat(kubernetes): expose ouroboros as a per-tenant addon
Adds the OuroborosAddon typedef, addons.ouroboros knob, and tenant HelmRelease that deploys the cozy-ouroboros chart inside a tenant Kubernetes cluster via the host's flux-helm-controller and the tenant admin-kubeconfig — same shape as the cert-manager addon. The HelmRelease overrides controller.mode back to coredns-import because tenants run the cozystack-coredns wrapper (which carries the `import /etc/coredns/custom/*.override` directive); the parent chart's default of mode=coredns is the right shape only for the Talos-managed host CoreDNS. controller.gatewayApi.enabled is mirrored from the per-tenant addons.gatewayAPI.enabled — forcing --gateway-api on a tenant without Gateway-API CRDs CrashLoops the controller on WaitForCacheSync against missing GVRs. dependsOn grows a conditional gateway-api-crds entry on the same gate. controller.clusterDomain is intentionally NOT pre-set by the wrapper. The chart auto-detects it inside the controller pod by parsing /etc/resolv.conf — the search list of which is `*.<ns>.svc.<domain>` and reflects exactly the zone the tenant CoreDNS actually serves (governed by KamajiControlPlane.spec.networkProfile.clusterDomain defaulting to `cluster.local`, and the cozystack-coredns wrapper kubernetes-plugin zone, also `cluster.local`). Pre-setting from a host-side fact like `_cluster.cluster-domain` would override the auto-detect with a value that does not match what the tenant kube-dns serves, silently emitting NXDOMAIN-bound rewrites. Operators that run a tenant on a non-default cluster-domain can still override via addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain. If addons.ouroboros.enabled is true but ingressNginx is off, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool true, string "true"/"True"/"TRUE", numeric 1, YAML "yes"/"Yes"/"YES"/"on"/"On"/"ON"/"y"/"Y") via toString-then-lower-then-string-compare; direct bool eq would error on type mismatch when an operator passes the value as a string from --set, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. The acknowledgeUncleanRemoval gate is defence-in-depth for the rare edge case where helm uninstall fails to land cleanly (tenant unreachable, hook timeout, manual kubectl delete bypassing helm) — in the normal path the upstream chart's pre-delete cleanup hook (coredns-cleanup-hook.yaml, vendored at 0.6.0) nulls the ouroboros.override key automatically when the HelmRelease is uninstalled. The gate only blocks render when a leftover HelmRelease on the host kube-apiserver shows the previous install is still sitting on the cluster. The component is also registered in the kubernetes-application PackageSource (as kubernetes-ouroboros at path system/ouroboros) so the tenant HelmRelease's ExternalArtifact resolves. make generate re-emitted README.md, values.schema.json, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. helm-unittest covers: render gating on enabled flag, hard-fail when ingress-nginx is off, namespaces and labels, chartRef, dependsOn cilium/coredns/ingress-nginx + conditional gateway-api-crds, default coredns-import mode, valuesOverride precedence, gateway-api flag mirroring in both directions, gateway-api truthy-spelling acceptance (including case-insensitive on/On/True/TRUE), the lookup-driven acknowledge gate (kubernetesProvider mock asserts the failedTemplate path when a prior HelmRelease still exists on the host), the lookup-returns-nil CI/dry-run path (render proceeds when the cluster mock has no objects), the _namespace.etcd-empty render gate, and that controller.clusterDomain is NOT pre-set in the rendered HelmRelease (so chart auto-detect runs). Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9f8e7f9d4a |
feat(coredns): import /etc/coredns/custom into the Corefile
Extends the cozystack-coredns wrapper with the standard CoreDNS `import /etc/coredns/custom/*.override` directive in the .:53 server block and renders an empty kube-system/coredns-custom ConfigMap as a chart-managed prerequisite, mounted into the CoreDNS Deployment at /etc/coredns/custom/ via extraVolumes / extraVolumeMounts. The ConfigMap carries helm.sh/resource-policy: keep so the chart-managed empty value cannot strip what later writers add into it. This exposes a stable extension point on the cozystack CoreDNS for operators (or other cozystack components) that need to add plugin-only rewrite snippets without touching the chart-rendered Corefile — otherwise the chart would re-template data.Corefile on every Flux reconcile and overwrite the additions. The same package serves both host and tenant CoreDNS deployments, so this plumbing covers both layers without per-layer duplication. The volume mount is `optional: true` for defence-in-depth: a future chart upgrade that drops the templates/coredns-custom.yaml render still leaves CoreDNS healthy with no override snippets in play. helm-unittest fixtures pin the ConfigMap shape (name, namespace, keep annotation, no data), the `import` directive in the rendered Corefile, the presence of the `reload` plugin (required for live ConfigMap updates), and the volume + volumeMount on the CoreDNS Deployment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
28e88e9b1d |
feat(ouroboros): add cozy-ouroboros system chart
New packages/system/ouroboros/ vendors lexfrei/ouroboros 0.6.0 from
oci://ghcr.io/lexfrei/charts/ouroboros and pins cozystack-side defaults
for a hairpin-NAT fix on top of ingress-nginx + PROXY-protocol.
The controller watches Ingress (and optionally Gateway / HTTPRoute)
for public hostnames and rewrites cluster-internal DNS so those
hostnames resolve to a small in-cluster TCP proxy that prepends the
PROXY-protocol v1 header before forwarding to ingress-nginx — without
that, intra-cluster traffic to the cluster's own public hostnames
arrives at ingress-nginx without the PROXY header it requires and
ingress-nginx closes the connection.
cozystack pins:
* controller.mode=coredns by default (mutates the live
kube-system/coredns Corefile in place between BEGIN/END markers).
The host CoreDNS on cozystack is Talos-managed and does not carry
an `import /etc/coredns/custom/*.override` directive, so the
alternative coredns-import mode would silently no-op there.
Tenant deployments override this back to coredns-import via the
apps/kubernetes wrapper (separate change).
* proxy.target.host=ingress-nginx-controller.cozy-ingress-nginx.svc.cluster.local —
cozystack deploys ingress-nginx into cozy-ingress-nginx in both
the host control plane and tenant clusters; the chart-default
`ingress-nginx` namespace exists nowhere in cozystack and would
silently dial a non-existent Service.
* controller.gatewayApi.enabled defaults to upstream's false at the
parent level. The bundle template (separate change) flips it on
for the host scope; the tenant wrapper mirrors per-tenant
addons.gatewayAPI.enabled. Forcing --gateway-api on a tenant
without Gateway-API CRDs CrashLoops the controller on
WaitForCacheSync.
* image is digest-pinned at
ghcr.io/lexfrei/ouroboros:0.6.0@sha256:b33c415cd2edf231eb62076297fee1b75657bef2bfcc0be916b84088aa49479d and pulled
directly from the upstream registry; this package is intentionally
not mirrored under ghcr.io/cozystack/*. Air-gapped operators
mirror both the chart and the image separately — bump together
with OUROBOROS_CHART_VERSION in the Makefile.
The vendored chart, helm-unittest fixtures (mode dispatch, RBAC
narrowing per mode, Gateway-API gating, target FQDN), and chart
layout follow the same shape as packages/system/cilium and
packages/system/metallb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov Daniil
|
cc7551cf1b |
test(e2e): use $name in kafka.bats kubectl wait (review feedback PR #2554)
Address lexfrei's nit on PR #2554: the pre-existing
`kubectl wait kafkas -n tenant-test test ...` on line 47 hardcoded the
resource name, while the existence backstop one line above was already
parametrized with `$name` in
|
||
|
Myasnikov Daniil
|
fe138afab7 |
test(e2e): add --ignore-not-found to vminstance.bats end-of-test deletes
Symmetrize the post-success cleanup on lines 83-84 with the pre-clean shape on lines 12-13/41. Without --ignore-not-found, a race that already removed the resource (e.g., a parallel teardown or finalizer-driven GC) turns a passing test into a failure after assertions have already run. Review feedback PR #2555 (lexfrei follow-up #1). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7ed6444108 |
feat(installer,e2e): cozystackOperator.helmReleaseInterval value + parallel post-install prep
Two complementary changes that depend on the operator's `--helmrelease-interval` flag landing via #2509: 1. Installer chart exposes `cozystackOperator.helmReleaseInterval` (default empty -> flag not rendered -> operator default 5m applies). E2E install passes `30s`. Eliminates the 8-10 min Flux dead zone where dependency-blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only every 5 min. 2. LINSTOR / StorageClass / MetalLB setup, previously sequential `@test` cases after the 15 m platform HR wait, now run as a background prep that completes during the platform reconcile. None of them gates platform HR reconcile, so they can move out of the critical path. Helper script `hack/e2e-post-install-prep.sh` polls for its prerequisites and is awaited at the end of the `Install Cozystack` test so failures still surface deterministically. Includes the T2-race fix: wait on the LINSTOR HR being `Ready` before polling for `deploy/linstor-controller`. The old 5-min hard timeout fired during cold installs where the LINSTOR HR's dependency chain takes longer than 5 min to resolve. Production defaults are unchanged. The 30 s interval is opt-in per install command. Depends on: - #2509 (operator-side `--helmrelease-interval` flag). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
myasnikovdaniil
|
7520967d8d |
feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) (#2509)
## Why
The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:
- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.
This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.
## What
### New operator flags
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.
### Chart exposure
`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.
## Production safety
- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)
## Verification
- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`
## Followups (deliberately not bundled)
- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.
## Test plan
- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.
* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.
* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
086e9bb35b |
fix(installer): bootstrap cozy-system via --create-namespace + label hook (#2508)
## Why helm v3 has a chicken-and-egg with charts that ship their own Namespace resource: - **With `--create-namespace`** on the install: helm pre-creates the namespace via plain kubectl-create (no helm meta annotations); the chart's own Namespace apply then fails with `already exists`. - **Without `--create-namespace`**: helm fails immediately because it can't write its release-secret to a non-existent namespace. The cozy-installer chart shipped a `Namespace cozy-system` resource since the chart's introduction (December 2025). Every cold install since has hit this conflict. It was masked by the 3× retry on `Install Cozystack` in `.github/workflows/pull-requests.yaml`: attempt 1 fails with the conflict, attempt 2 sees the existing failed release and takes the upgrade code path which patch-merges instead of strict-create. Surfaced when retries on the install step were dropped. ## What - Remove `Namespace cozy-system` from `packages/core/installer/templates/cozystack-operator.yaml`. - Add `packages/core/installer/templates/cozy-system-labels.yaml`: a pre-install/pre-upgrade hook (ServiceAccount + ClusterRole + ClusterRoleBinding + Job) that patches `cozystack.io/system=true` and `pod-security.kubernetes.io/enforce=privileged` onto the namespace. - Hook RBAC is scoped to `verbs: [get, patch]` on `resourceNames: [cozy-system]` only. - `hook-delete-policy: before-hook-creation,hook-succeeded` so the hook surface only exists during install/upgrade. Install commands now use `helm upgrade --install --namespace cozy-system --create-namespace …` — standard pattern, matches kube-prometheus-stack / argo-cd / cert-manager. ## Verification End-to-end on a kind cluster: - Cold install: 3.3s clean, no retries needed - Upgrade (re-run): 3.0s, idempotent, namespace stays labeled - Helm uninstall + re-install: clean ## Test plan - [ ] CI on this PR passes E2E - [ ] Existing clusters: namespace already labeled correctly, hook re-asserts on next upgrade — no migration needed ```release-note fix(installer): cold installs no longer fail on the chart's Namespace resource colliding with `helm --create-namespace`; cozy-system is now bootstrapped via a pre-install hook. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an automated pre-install/pre-upgrade step that applies system identity and pod-security labels to the system namespace during deployment, run in a hardened one-shot task before normal workloads start. * **Chores** * Separated namespace initialization from the operator install flow so the operator no longer renders or retains the system namespace directly, improving installation sequencing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
25180c58f7 |
fix(virtual-machine): bump HR readiness wait from 5s to 60s
5s was too tight under runner load — both vm-disk-$name and vm-instance-$name HR waits could miss the operator's reconcile window. Aligns with the other timeout bumps in this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
12ccc53d9d |
test(e2e): tighten event-driven backstops
- kafka.bats: use $name in the kafkas existence backstop, matching the variable used in the kubectl wait below it (consistency with the rest of this PR's backstops). - e2e-install-cozystack.bats: the Tenant→HelmRelease materialisation is asynchronous (cozystack-controller observes the Tenant CR and creates the HR), so guard hr/tenant-test with the same existence backstop pattern used elsewhere. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
90b3719f6a |
fix(mongodb): expose replica-set members per pod for external access (#2514) (#2522)
## What this PR does Fixes #2514. When `external: true` is set on a non-sharded `MongoDB`, Cozystack rendered a single `LoadBalancer` Service whose selector matched **all** replica-set members (`app.kubernetes.io/replset: rs0`). Round-robin balancing sent ~⅔ of writes to secondaries and they were rejected with `MongoServerError: not primary`. Drivers couldn't recover on their own either: connecting through the LB returned the in-cluster DNS names of all members from `isMaster`, none of which resolve from outside the cluster, so automatic primary discovery failed. This PR replaces the manual `external-svc.yaml` (for replica-set mode) with the Percona MongoDB operator's native `replsets[].expose` feature: ```yaml replsets: - name: rs0 expose: enabled: true type: LoadBalancer externalTrafficPolicy: Local ``` The operator then: - creates one `LoadBalancer` Service **per pod** (`<release>-rs0-0`, `-1`, `-2`) with `statefulset.kubernetes.io/pod-name` selectors, - watches `Service.status.loadBalancer.ingress` for each pod and rewrites the member `host` in `rs.conf` to the external IPs. External clients use a standard replica-set URI (`mongodb://...:27017,...:27017,...:27017/?replicaSet=rs0`) and the driver discovers the primary correctly from outside the cluster. For **sharded** mode the existing single `LoadBalancer` is preserved — it front-ends stateless `mongos`, which can route writes to the right shard primary itself. `external-svc.yaml` is now scoped to that case only. ### Tested - `helm unittest .` in `packages/apps/mongodb/`: 91/91 pass. - Applied on a dev stand: - `replsets[0].expose` is set as expected in the rendered PSMDB CR. - The Percona operator (`v1.21.1`) accepts the spec and creates 3 per-pod `LoadBalancer` Services with `statefulset.kubernetes.io/pod-name` selectors. The previous single round-robin Service is no longer rendered. - With an LB IP pool present, the operator promotes the allocated LB IPs into the PSMDB `status.host` field and into `rs.conf`. End-to-end mongosh write loop wasn't run on the dev stand because the underlying network does not advertise/route allocated LoadBalancer IPs (no working L2/BGP path), so the rewritten `rs.conf` members are unreachable in that environment. The structural change is otherwise fully verified and the issue's reproduction scenario expects a real LB-capable network. ### Release note ```release-note fix(mongodb): expose each replica-set member through its own LoadBalancer when `external: true`, so external clients can discover the primary via the standard replica-set connection string and writes no longer fail with "not primary" errors. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * External LoadBalancer services now only render when both external and sharding are enabled. * Replica set external exposure is now conditional based on the external setting. * Service configuration now uses LoadBalancer type with Local traffic policy for external access. * Service selectors refined for proper mongos targeting in sharded deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
0b70197bf1 |
fix(kafka): bump default resourcesPreset to medium (#2537)
## What this PR does Changes the default `resourcesPreset` for both `kafka` and `zookeeper` in the Kafka managed application from `small` (1 CPU / 512Mi) to `medium` (1 CPU / 1Gi). The `small` preset is no longer sufficient for the current Strimzi 0.45 / Kafka 3.9 image baseline: - Strimzi sets JVM heap to 50% of the container memory limit for Kafka and 75% for ZooKeeper. With a 512Mi limit that leaves only 256Mi (Kafka) or 128Mi (ZooKeeper) for the JVM non-heap. - Actual non-heap usage in the Kafka 3.9 image (Metaspace, code cache, direct buffers, JMX javaagent, embedded Jetty admin server for `/v1/ready` and KRaft migration endpoints, thread stacks) systematically exceeds those budgets at startup, triggering a cgroup OOM kill (exit 137) before the broker can connect to ZooKeeper. Result on a fresh deployment with the previous default: every Kafka broker and every ZooKeeper pod ends up in `CrashLoopBackOff`, with `lastState.terminated.reason: OOMKilled`. ZooKeeper occasionally hits `OOMKilled` once quorum traffic ramps up, breaking the quorum and causing Kafka brokers to fail with `ZooKeeperClientTimeoutException`. The `medium` preset (1Gi) is the smallest preset that gives both Kafka (heap 512Mi + ~500Mi non-heap budget) and ZooKeeper (heap 768Mi + 256Mi non-heap budget) enough headroom to start cleanly. Users who need more can still pick `large`, `xlarge`, or `2xlarge`, or override `resources` explicitly. The `small` preset is intentionally kept in the enum so existing deployments that pin `resourcesPreset: small` are not broken; only the default changes. ### Release note ```release-note fix(kafka): default `resourcesPreset` for both Kafka and ZooKeeper is now `medium` (1 CPU / 1Gi). The previous default (`small`, 512Mi) is no longer sufficient under the current Strimzi 0.45 / Kafka 3.9 image and caused OOMKilled CrashLoopBackOff on fresh deployments. Existing deployments that explicitly set `resourcesPreset` are unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated default resource allocation preset from "small" to "medium" for Kafka and ZooKeeper components across CRD type definitions, Helm chart values, and JSON schema configurations. * **Documentation** * Updated parameter reference tables to reflect the new default resource preset values for both Kafka and ZooKeeper. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
0a3a5039d4 |
fix(kubevirt-operator): move PrometheusRule to Helm-rendered template (#2543)
## What this PR does
Move `alerts/PrometheusRule.yaml` from raw-file loading (via
`.Files.Glob` in
`templates/alerts.yaml`) to a proper Helm-rendered template
`templates/prometheusrule.yaml`.
The raw-file approach silently passed unescaped `{{ $labels.xxx }}` to
Kubernetes unchanged, which worked by accident. As a Helm template,
those
expressions must be escaped with `{{ "{{" }}` to prevent Helm from
trying
to evaluate them as Go template variables.
Also removes `templates/alerts.yaml` which became dead code once the
`alerts/` directory is empty.
### Release note
\`\`\`release-note
fix(kubevirt-operator): move PrometheusRule alerts to Helm-rendered
template and escape Prometheus template variables
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Reorganized alert configuration structure for VM and VMI monitoring.
Alerts that trigger when virtual machines or machine instances remain in
a non-running state for 10 minutes are now managed through an updated
template-based approach, consolidating alert definitions while
maintaining the same monitoring behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
87f2f64414 |
chore(kubevirt-operator): update to v1.8.2 (#2502)
## What this PR does Updates KubeVirt operator from v1.6.3 to v1.8.2. This update brings: - New CRD fields for changed block tracking (required for storage-agnostic backups) - Architecture support updates (s390x support, ppc64le deprecated) - Updated operator manifest with latest capabilities The standalone ServiceMonitor and alerts templates were removed as they are now integrated into the main operator manifest. ### Screenshots N/A - infrastructure component update ### Release note \`\`\`release-note chore(kubevirt-operator): update to v1.8.2 Updated KubeVirt operator from v1.6.3 to v1.8.2, adding support for changed block tracking and latest architectural improvements. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new configuration fields for changed block tracking, feature gate management, hypervisors, and VM backups. * **Chores** * Upgraded KubeVirt operator to v1.8.2. * Updated RBAC permissions and cluster access policies. * Removed Prometheus monitoring integration and alert template processing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
7d292eb187 |
chore(e2e): drop seaweedfs/monitoring flux-reconcile workaround
With Strategy=RetryOnFailure + RetryInterval=30s plus the bats --set cozystackOperator.helmReleaseInterval=30s override, the existing 2m wait converges without the manual flux reconcile --force band-aid. Removing the workaround turns CI itself into the test of the fix. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
730037c85d |
fix(installer): address review feedback on cozy-system bootstrap
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks. - Skip hostNetwork=true and not-ready/CNI tolerations on the hosted variant: managed clusters provide CNI before install, and kube-system may carry strict PSA there that would reject hostNetwork pods. - Add packages/core/installer/templates/cozy-system-namespace.yaml gated on .Values.bareNamespace (default false). The top-level Makefile's manifests target now passes --set bareNamespace=true and renders Namespace + operator into _out/assets/cozystack-operator-*.yaml so the kubectl-apply install path stays self-contained. Helm install/upgrade callers keep the default and bootstrap via --create-namespace + the pre-install labeler hook. - Add a bats assertion in hack/e2e-install-cozystack.bats checking cozy-system carries pod-security.kubernetes.io/enforce=privileged and cozystack.io/system=true after helm install. - Inline comment near hostNetwork=true documenting why dnsPolicy left at Default (cluster DNS not up at install time; ClusterFirstWithHostNet would not help). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
929c623af1 |
test(operator): assert Install/Upgrade Remediation stays nil
helm-controller's XValidation rule rejects Strategy.Name=RetryOnFailure
with RetryInterval set alongside a Remediation entry, so a future
"for safety" reintroduction of Remediation{Retries: -1} would silently
break every HR. Pin the deliberate omission.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
d0e4b4dbe5 |
refactor(operator): extract parsePositiveDuration into testable helper
The previous closure inside main() couldn't be tested because every error
path called os.Exit(1). Lifting it to a package-level function returning
(time.Duration, error) lets a table test exercise happy paths and rejections
("0s", "-5m", "5x", empty input) directly. main() retains the os.Exit
behaviour via a thin wrapper.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
266465b410 |
fix(operator): copy MaxHistory to local before taking address
Avoid pointer aliasing across all generated HelmReleases. r.HelmReleaseMaxHistory is set once at startup today, but a future mutation through the field would silently affect every in-flight HelmReleaseSpec. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
d9023c9b15 |
feat(cozyreport): collect Flux/cert-manager/host context + summary.txt (#2553)
## What this PR does Makes `cozyreport.tgz` actually useful when triaging an E2E failure deeper than a single pod log. Adds: - Flux controller logs (helm-controller, source-controller, notification-controller, kustomize-controller, last 2000 lines each) - Flux source resources (`HelmRepository`, `OCIRepository`, `GitRepository`, `ExternalArtifact`) - Decoded Helm storage secrets for non-Ready HRs (`base64 -d | base64 -d | gzip -d`) - Cluster events (all + warning-only filtered file) - cert-manager `Certificate` / `CertificateRequest` / `Order` / `Challenge` resources + cert-manager logs - `cozystack-operator` deployment logs (current + previous) - `Application` / `ApplicationDefinition` / `Tenant` resources - Sandbox host context per node: `df`, `free`, `ps`, `dmesg`, `talosctl logs/dmesg/kubelet/containerd` New executable `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive root listing what is broken right now — the first thing to read when downloading the artifact from a CI failure. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a timestamped diagnostic summary report that highlights failing resources and key cluster issues. * Expanded diagnostic collection: controller logs (current+previous), Flux source status, cert-manager certificate details, Helm release payloads, and Cozystack resource status. * Added host-level diagnostics (disk/memory/process/dmesg) and improved event/warning listings to aid troubleshooting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
36dbd6b27c |
test(tests): pre-pull timing-sensitive platform images before install (#2556)
## What this PR does Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server) fail when replicas start at materially different times due to image-pull stagger across nodes. Adds a DaemonSet-based pre-pull step that runs before `helm install`, ensuring every node has the images cached so replicas start within milliseconds of each other. `hack/e2e-prepull-images.sh` accepts image refs on stdin and creates one container per image so pulls run in parallel — total time = max(any single image) rather than sum. The bats test sources image refs directly from the rendered platform charts via `yq`, walking only PodSpec-shaped objects, so version bumps stay in sync without a hardcoded list. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added an end-to-end test that pre-pulls platform container images before the installation workflow to improve install reliability. * **Chores** * Added an automated pre-pull utility that reads image lists, deduplicates and sorts them, deploys a short-lived pre-puller across nodes, waits for completion, then cleans up. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
6ad5bf0bbf |
fix(objectstorage-controller): retry on Bucket conflict during BucketAccess reconcile (#2529)
## What this PR does
Carries an upstream-bound patch against COSI v0.2.2's BucketAccess
sidecar reconciler so it survives the optimistic-concurrency race with
the central Bucket reconciler running in the same process.
In `sidecar/pkg/bucketaccess/bucketaccess_controller.go`, the `Add` path
does:
```go
bucket, _ := buckets().Get(...)
// ...
if controllerutil.AddFinalizer(bucket, BABucketFinalizer) {
buckets().Update(ctx, bucket, ...) // <-- 409 here
}
```
Between `Get` and `Update` the Bucket reconciler in the same binary can
mutate the object and bump `resourceVersion`, producing:
```
Operation cannot be fulfilled on buckets.objectstorage.k8s.io "bucket-...":
the object has been modified; please apply your changes to the latest
version and try again
```
surfaced to users as a `FailedGrantAccess` event on the BucketAccess.
This patch wraps the Bucket finalizer add in
`retry.RetryOnConflict(retry.DefaultRetry, ...)` and re-`Get`s the
Bucket inside the closure so each retry mutates the freshest version.
Patch is held as `91-bucketaccess-conflict-retry.diff` and consumed at
image build time (re-introduces `git` to the source stage and the `COPY
patches /patches` + `git apply /patches/*.diff` steps that existed
before `c29d501b` dropped them when 89/90 were upstreamed). The
convention is identical: drop the local patch the moment the upstream PR
merges and ships in a tagged release.
### Release note
~~~release-note
[objectstorage-controller] Carry a patch that retries on Bucket update
conflict during BucketAccess reconcile, eliminating the "object has been
modified" FailedGrantAccess event seen in CI.
~~~
## Test plan
- [x] `make image` for `system/objectstorage-controller` and verify
`values.yaml` picks up the new digest.
- [x] Re-run the cozystack-e2e suite that emitted the
`FailedGrantAccess: ... object has been modified` event on
`tenant-test/BucketAccess/bucket-test-admin` and confirm it no longer
fires.
- [ ] Verify `kubectl describe bucketaccess` reports
`AccessGranted=true` on first reconcile, no requeue from the conflict
path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced bucket access update reliability with improved conflict
handling during concurrent operations. The system now automatically
retries operations and refetches data to prevent update conflicts.
* **Chores**
* Improved Docker build process with automatic patch application support
for source customization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
8fbbef3f02 |
fix(harbor): drive bucket-secret.yaml from values, gate HR on BucketInfo (#2528)
## Summary - `cozy-harbor/templates/bucket-secret.yaml` no longer relies on `lookup` against the BucketAccess credentials Secret. It is now driven by `.Values.bucket.bucketInfo` (a JSON string) with `dig`-based safe accessors, so a missing/empty/partial value renders nothing rather than crashing with `index of untyped nil` (the failure mode that broke harbor in CI #25049445125). - The downstream `<release>-system` HelmRelease now sources `BucketInfo` via `valuesFrom` (`valuesKey: BucketInfo`, `targetPath: bucket.bucketInfo`). With the default `optional: false`, helm-controller refuses to compose values until the COSI BucketAccess controller has populated the Secret — both gating initial reconciliation and forcing a config-digest change (and thus a helm upgrade) when the credentials arrive. Flux HR `dependsOn` cannot reference COSI resources directly, so `valuesFrom` is the correct primitive here. - `bucket.secretName` was the only consumer of the old `lookup` and is removed from both the system chart's `values.yaml` default and the apps chart's `values:` block. ## Why this shape helm-controller's upgrade trigger is digest-based over composed values + chart artifact; a `lookup` returning new data on a later reconcile is not enough to force an upgrade on its own. Routing `BucketInfo` through `valuesFrom` puts the bucket data on the digest path, so the chart re-applies the moment credentials become available. ## Test plan - [x] `helm template` of `packages/system/harbor` with `bucket.bucketInfo` unset, empty string, empty JSON object `{}`, and fully populated — only the populated case renders the `*-registry-s3` Secret; the others render nothing without erroring. - [x] `helm template` of `packages/apps/harbor` produces the expected `<release>-system` HR with the new `valuesFrom` entry and the cleaned-up `values:` block. - [ ] Re-run the harbor E2E on a fresh stand and confirm 15/15. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Harbor bucket config now comes from a centralized Secret merged into chart values; in-values bucket entry removed. * **Bug Fixes** * Deployment synchronization and Helm upgrade behavior improved by depending on the Secret’s readiness/contents. * **Tests** * Added unit tests covering when the bucket secret is rendered (or not) and a make target to run those tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |