mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
ci/parallel-container-disk
635
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 cozystack-ci[bot]
|
7cc5df5139 |
Prepare release v1.5.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
0240dc128f |
fix(platform): order backupstrategy-controller after cozystack-basics
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
cozystack-ci[bot]
|
22d9f224f9 |
Prepare release v1.5.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
e1aaa90ed3 |
Prepare release v1.5.0
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Myasnikov DaniilandClaude
|
dc9427285d |
Merge branch 'main' into feat/flux-shard-operator-impl
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
aeaedca4a4 |
ci: consolidated platform & e2e stabilization batch (#2948)
## What Consolidated landing branch for a set of interdependent platform/CI stabilization fixes. It began as the source PRs in the table below (in dependency order) and has since grown with review-driven fixes and a few production-behavior changes — the full current inventory is under **Scope** below. Landing them as one unit lets CI run against the real combined state instead of a fragile bottom-up merge train. ## Source PRs (dependency order) | # | Commit | Source PR | Addresses | |---|--------|-----------|-----------| | 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi crashloop during cert provisioning | | 2 | test(e2e): bucket.bats port-forward + S3 client reliability | #2944 | flaky bucket test | | 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop volumeSizeLimitMB | #2943 | missing bucket/access classes | | 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps) | #2874 | cilium "IP already in use" leak | | 5 | refactor(build): mode=max registry cache + main-only warmer | #2938 | build cache | | 6 | ci(build): isolate each PR build on its own ephemeral runner VM | #2939 | shared-buildkit contention | | 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline | #2928 | LINSTOR provisioning timeouts | | 8 | test(metallb): assert digest-pinned image form, not version literal | #2873 | brittle metallb assertion | ## Why consolidated These form a dependency DAG (verified from CI logs): e.g. 2943 needs 2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested bottom-up, each lower PR runs with none of the fixes above it and can't go green alone. This branch carries all of them, so CI runs against the real combined state. ## Verification status CI is green on the latest head (`5653c30`): full-suite E2E passes end-to-end and is reproducible (2 of 3 runs on this SHA green). The one red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the sandbox, unrelated to the diff — a different environmental subsystem failed each run (details in the comments below). The earlier 3-hour crust-gather snapshot hang is fixed in `5653c30`. ## Scope beyond the original 8 The branch has grown past the 8 source PRs above with review-driven fixes and a few production-behavior changes. For reviewer transparency, the full set: **Review fixes (@lexfrei review):** - **B1 (blocker)** — `8f41910` converts the SeaweedFS `-lock`/`-readonly` BucketClasses and the s3 service-name override into `patches/`, wired into `make update`, so a re-vendor no longer drops them. - **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods (both delete remedies were RBAC-forbidden). - **FU3** — `014de3f` adds a unit test for the HelmRelease update conflict-retry path. - **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any non-terminal owner phase, not just `Running`. - **FU7** — `f3914d2` sets `persist-credentials: false` on the build-cache checkout (SHA-pinning is handled repo-wide in #2849). **Production-behavior changes (not in the table above):** - VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring (`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops install-time eviction churn. - cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease update path (`728f676`). **E2E robustness follow-ups:** per-test crust-gather snapshots (`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget (`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin (`95f42aa`). **Not addressed (by decision):** FU2 — the s3 Service keeps `seaweedfs.name`-based naming via patch (rendered output unchanged; cozystack renders `fullname == seaweedfs`, so it matches the `componentName`/`fullname` siblings in practice). FU6 — fork PR build push is handled by the existing mirror-to-same-repo-branch workflow. ## Relates to #2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those can be closed; otherwise they remain the granular per-PR review path. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added CI build cache warming for main branch builds. * Added a best-effort in-cluster Cilium endpoint self-heal watchdog for e2e installs. * Extended SeaweedFS COSI with object-lock support plus separate readonly access. * **Improvements** * Reduced e2e install flakiness with a shared readiness deadline, safer waits, and improved cleanup/diagnostics capture. * Prevented install-time churn by switching VPA update mode to **Initial**. * Added container startup probes and updated SeaweedFS volume sizing. * **Tests** * Improved e2e robustness (etcd backup schedule waiting, digest-pinned image checks, and corrected S3 port-forwarding). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
95f42aac31 |
fix(e2e-sandbox): pin crust-gather to v0.15.2 (tag needs leading v)
The crust-gather install.sh substitutes the version verbatim into both the release-tag path and the asset filename: releases/download/<ver>/kubectl-crust-gather_<ver>_<target>.tar.gz The release tag is 'v0.15.2', so passing '0.15.2' 404s on the tag path (curl exit 22) and fails the image build. The 'v0.15.2' asset variant exists too, so the leading v satisfies both. Verified HTTP 200. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
50e5f94132 |
test(e2e): per-test crust-gather snapshots + cleanup-on-exit via cozytest.sh
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined. run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook. Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
d3958c5bcb |
refactor(build): mode=max registry cache with a main-only warmer (#2938)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei KvapilandClaude
|
7f994f2d75 |
fix(platform): avoid SIGPIPE in migration 44 drain check
head -n 1 closing the pipe early under pipefail aborts the migration once the HelmRelease name list outgrows the pipe buffer; count with wc -l instead. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
6e219c241b |
feat(platform,gateway): distribute operator wildcard secret name
Add publishing.certificates.wildcardSecretName to the platform chart and propagate it through the cozystack-values channel as _cluster.wildcard-secret-name. Only the Secret NAME travels — the cert/key material is never broadcast, unlike the public kube-root-ca that already rides this channel. When the key is set, the gateway chart switches the rendered TenantGateway to certMode=existingSecret with a wildcardSecretRef pointing at the operator-supplied Secret, and skips the ACME solver/provider/issuer validations that no longer apply. Wildcard mode takes precedence over solver=dns01. TLS-passthrough services and attached namespaces continue to render unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1d747b3d8d |
feat(platform): auto-wire KubeVirt permittedHostDevices and HostDevices feature gate when gpu-operator is enabled (#2768)
Closes #2765. ## What this PR does Makes the passthrough and vGPU variants of `cozystack.gpu-operator` produce a KubeVirt CR that is immediately usable, without an out-of-band `kubectl patch`. Today the gpu-operator stack reports green, `nvidia.com/<model>` shows up under node `.status.allocatable`, but every `VirtualMachine` referencing the resource via `domain.devices.hostDevices` is rejected by the admission webhook because `HostDevices` is not in `developerConfiguration.featureGates` and the resource is not listed under `permittedHostDevices`. That manual reconciliation is now driven by the platform. The change has three parts. `packages/system/kubevirt` now parameterizes the KubeVirt CR template. `developerConfiguration.featureGates` keeps its existing static defaults and appends `.Values.extraFeatureGates`; `spec.configuration.permittedHostDevices` renders verbatim from `.Values.permittedHostDevices` when set; `spec.configuration.mediatedDevicesConfiguration` does the same. Defaults stay empty so the rendered CR is byte-identical to the previous template when no overrides are supplied. The chart's `make update` target re-applies the four sed transforms after fetching the upstream CR, so refreshing the KubeVirt release version stays a single command. `packages/core/platform` adds two defaults tables under `files/`. `gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName` slugs follow the `<arch>_<model>_<form>_<mem>` convention `nvidia-sandbox-device-plugin` generates as of v25.x, and `externalResourceProvider: true` is set everywhere because those resources are advertised by the sandbox plugin rather than KubeVirt's in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q, V100D-32C, T4-16Q) — the same family span the upstream `vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and Blackwell SR-IOV vGPU is intentionally out of scope for the default table and is expected to be advertised via the user-override hook described below. `templates/bundles/iaas.yaml` now selects between the two tables via a new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough, `vgpu` for mediated; default is passthrough). The same variant flows through to the gpu-operator Package CR, replacing the previous always-default call. When the package is enabled, the platform injects `HostDevices` into `extraFeatureGates` plus the chosen defaults into `permittedHostDevices` for the kubevirt component's values. Operators extend or replace via top-level platform values: ```yaml gpu: replaceDefaults: false # append (default) vs. wipe-and-replace permittedHostDevices: pciHostDevices: [] # cluster-specific PCI passthrough entries mediatedDevices: [] # cluster-specific vGPU entries mediatedDevicesConfiguration: {} # extra mediated-device types / per-node mappings ``` `docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step and describe the auto-wiring contract, the per-variant defaults, the escape hatches, and the sandbox-device-plugin `resourceName` slug convention so an operator whose node publishes a different name knows where to override. ## Test coverage `helm-unittest` covers eight branches in `packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`: gpu-operator disabled emits no extras; gpu-operator enabled at default variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user overrides append by default; `replaceDefaults: true` wipes the platform table; an unknown variant fails fast with a descriptive error. `packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the empty-values render and the three populated shapes at the CR template level. `make helm-unit-tests` passes locally on the full repo. ### Release note ```release-note feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults. **Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Bundle-managed GPU Operator with selectable passthrough vs vGPU variants; platform now injects GPU settings into KubeVirt and forwards vGPU manager overrides when enabled. * Default platform mappings for PCI passthrough and vGPU mediated-device resources added. * **Documentation** * Expanded GPU/vGPU deployment guide with bundle-managed vs manual guidance, wiring/precedence rules, resource-name conventions, and verification steps. * **Tests** * New and expanded tests for bundle→KubeVirt wiring, variant behaviors, default/override merging, and idempotent template parameterization. * **Chores** * Added Helm test target and hardened KubeVirt template update/idempotency checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
b6108039b7 |
feat(gpu-operator): add container variant for preinstalled host driver (#2766)
## What this PR does Add a third architectural variant to `cozystack.gpu-operator` for hosts where the NVIDIA driver and `nvidia-container-toolkit` are already installed by the distro package manager (the typical Ubuntu/Debian GPU node shape: apt-installed driver + apt-installed CT). The existing variants do not produce a working setup in that shape. `default` (passthrough) runs `vfioManager`, which unbinds the host driver to bind `vfio-pci` — with the host driver active, `k8s-driver-manager` early-exits and the bind never happens, so the GPU never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host driver and a license server; the user-installed open or proprietary driver does not satisfy it. The new `container` variant maps onto upstream gpu-operator's third mode (`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with every component that would conflict with the host install switched off: - `driver.enabled=false` — use the host driver. - `toolkit.enabled=false` — host already has `nvidia-container-toolkit`. Enabling the operator's toolkit DaemonSet would overwrite the `/etc/containerd/config.toml` the host package configured (via `nvidia-ctk runtime configure`), breaking the host runtime wiring. Matches NVIDIA's documented guidance and the existing `examples/values-native-talos.yaml` reference, which pins the same flag for the same reason. - `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet. - `vfioManager.enabled=false` — do not unbind the host driver. Upstream renders this `true` unconditionally; it only stays at zero pods because no node carries a `vm-passthrough` sandbox assignment when `sandboxWorkloads` is off — pinned off so the rendered policy matches intent. - `cdi.enabled=false` — upstream defaults this `true`, but CDI spec generation is serviced by the operator-managed toolkit DaemonSet this variant disables. An apt host doesn't auto-generate CDI specs (`nvidia-ctk cdi generate` is a manual step) and uses the classic `nvidia-container-runtime` hook, so leaving CDI on would point the device plugin at CDI injection with no specs on the host and silently break allocation. - `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs. With this off, upstream `defaultWorkload` is already `container`, so no override is needed. Host prerequisite: because the operator's toolkit component is disabled, the node's containerd must already have the `nvidia` runtime registered (`nvidia-ctk runtime configure --runtime=containerd`); `apt install nvidia-container-toolkit` installs binaries only and does not wire containerd. With `driver.enabled=false` the operator uses the pre-installed host driver at its standard location, so a stock apt install needs no `hostPaths.driverInstallDir` override (Talos lands the driver under a non-standard prefix — see `examples/values-native-talos.yaml`). The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`) already enables `cozystack.gpu-operator` as an optional package — no bundle changes required. Users opt into this variant by setting `variant: container` in their `Package` CR. Verified locally with `helm template` against `values.yaml` + `values-container.yaml`: ClusterPolicy renders with `driver.enabled=false`, `toolkit.enabled=false`, `devicePlugin.enabled=true`, `vfioManager.enabled=false`, `cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm unittest` and the `gpu-operator` variant bats suite are green. Not yet smoke-tested on a real Ubuntu GPU node — the config matches the documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI off), but an end-to-end CUDA-pod run on hardware is still owed. Closes #2764 ### Release note ```release-note feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a "container" GPU operator variant to support containerized GPU workloads on hosts with the NVIDIA driver and nvidia-container-toolkit. * **Documentation** * Updated variants documentation and OS support table to include the container variant and per-host requirements (including Ubuntu and Talos notes). * Clarified Talos native pod example wording. * **Tests** * Added regression tests to verify container-variant configuration and component enablement/disablement. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
931f286b9f |
fix(ingress,platform): deliver publishing.proxyProtocol to host ingress-nginx (#2799)
## What this PR does `publishing.proxyProtocol: true` is documented to enable PROXY-protocol on the host ingress-nginx, but the three nginx config keys (`use-proxy-protocol`, `real-ip-header`, `enable-real-ip`) never reached the controller. They were written into the `ingress-system` component values of the `cozystack.ingress-application` Package, but that component has no `install:` block, so the package controller never builds a HelmRelease for it and the values are dropped. External traffic behind a PROXY-protocol L4 load balancer kept breaking. This routes the switch through `_cluster.proxy-protocol` in the `cozystack-values` Secret — the same path `expose-external-ips` already travels to reach the host ingress release — and the `extra/ingress` chart applies it to the external-facing ingress (the namespace named by `expose-ingress`). The dead component-values injection is removed. The `extra/ingress` chart also gains a first-class `proxyProtocol` value, so PROXY-protocol can be enabled per Ingress application (a per-app boolean knob like `cloudflareProxy`). Two `fail` guards keep the new knobs safe: the per-app `proxyProtocol` value is rejected on the external-facing (host) ingress — that path must go through `publishing.proxyProtocol`, which also installs ouroboros (hairpin handling) and arms the disable-hazard guard — and `cloudflareProxy` + `proxyProtocol` together are refused, since Cloudflare's HTTP proxy does not speak PROXY-protocol to origins and its `real_ip_header` server-snippet would override `real-ip-header: proxy_protocol`. `use-forwarded-headers` / `compute-full-forwarded-for` are deliberately not set: without a paired `proxy-real-ip-cidr` they would let clients spoof `X-Forwarded-For`, and the PROXY-protocol frame is already the source of truth for the client address. helm-unittest coverage pins runtime delivery on the host path, the per-app path, and that non-host tenants stay untouched; the previous test that only checked template emission into the (non-consumed) Package component is replaced. Affected versions: v1.4.0–v1.4.2 and current main (introduced in #2561). **Upgrade note (action required).** Because the flag was a silent no-op before this fix, a v1.4.x cluster that set `publishing.proxyProtocol: true` aspirationally — without its L4 load balancer actually injecting PROXY frames — kept working by accident. After this fix the flag really enables `use-proxy-protocol`, which makes ingress-nginx require a PROXY frame on every connection; if the LB does not send one, all external ingress breaks. Before upgrading, confirm the upstream LB injects PROXY-protocol v1 frames, or unset `publishing.proxyProtocol`. Closes #2796 ### Release note ```release-note fix(ingress): publishing.proxyProtocol now enables PROXY-protocol on the host ingress-nginx (the config was silently dropped before reaching the controller); the Ingress application also gains a proxyProtocol value to enable PROXY-protocol per-app. ACTION REQUIRED on upgrade: the flag was previously a no-op, so a cluster that set it without the upstream LB injecting PROXY-protocol v1 frames will lose all external ingress once it takes effect — verify the LB, or unset publishing.proxyProtocol before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tenant ingress support for PROXY-protocol via a new proxyProtocol option (default: false). * Platform-level publishing.proxyProtocol now controls external/host ingress behavior. * **Bug Fixes / Validation** * Rejects enabling per-app proxyProtocol on external-facing host ingresses. * Rejects using proxyProtocol together with cloudflareProxy to prevent conflicting real‑IP handling. * **Documentation** * Updated docs describing proxyProtocol usage, defaults, and safety requirements. * **Tests** * Added tests covering proxyProtocol wiring, config emission, and failure cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
479929ebfb |
ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown (#2558)
## What this PR does
Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.
On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.
> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (
|
||
|
Aleksei Sviridkin
|
c32812ccf9 |
fix(platform): deliver publishing.proxyProtocol to host ingress-nginx
The host PROXY-protocol switch wrote the nginx config keys into the ingress-system component values of the cozystack.ingress-application Package. That component has no install block, so the package controller never built a HelmRelease for it and the values were dropped — the host ingress-nginx never received use-proxy-protocol and external traffic behind a PROXY-protocol L4 load balancer kept breaking. Route the switch through _cluster.proxy-protocol in the cozystack-values Secret instead, the same path expose-external-ips travels to reach the host ingress release, and drop the dead component-values injection. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
9868c43e53 |
fix(build): read --cache-from from ghcr :latest so PR builds warm-start
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold. Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
117d317fd8 |
fix(platform): drop fragile numeric vGPU mediatedDeviceTypes defaults
The vgpu defaults shipped a mediatedDevicesConfiguration block whose mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not stable identifiers — they are the per-(SKU, driver) indices under which a host enumerates mdev types in /sys/.../mdev_supported_types/, so the same profile gets a different number on a different card or driver branch. Auto-injecting them made KubeVirt try to create mdev types that usually do not exist on the operator's nodes, and duplicated work the GPU Operator's vGPU Device Manager already does. Delegate mdev creation to the vGPU Device Manager, which is name-based and stable, and keep only the name-based permittedHostDevices.mediatedDevices exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation stays available through the .gpu.mediatedDevicesConfiguration override for operators who supply host-verified type ids; document the lookup recipe. Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and the tests accordingly (the default vgpu path now emits no mediatedDevicesConfiguration; the override path is covered explicitly). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
919073601d |
fix(platform): match NVIDIA passthrough defaults to the resourceName the plugin advertises
The auto-injected permittedHostDevices only works if each resourceName equals what nvidia-sandbox-device-plugin advertises on the node; otherwise the admission webhook rejects the VM — the very manual step this bundle removes. Several defaults did not match the slug the plugin (nvcr.io/nvidia/kubevirt-gpu-device-plugin) derives from the PCI-IDs database name, which uppercases the name and keeps the die suffix, the Tesla brand and the form-factor/memory tokens: 10DE:1EB8 TU104GL_T4 -> TU104GL_TESLA_T4 10DE:20B7 GA100_A30 -> GA100GL_A30_PCIE 10DE:2331 GH100_H100_PCIE_80GB-> GH100_H100_PCIE 10DE:2321 GH100_H100_NVL_94GB -> GH100_H100L_94GB 10DE:1DB1/1DB4/1DF6 + new 1DB5 -> GV100GL_TESLA_V100_* family Device 10DE:1DB1 was also mislabelled: it is the SXM2 16GB part, not the 32GB one — the 32GB SXM2 V100 is a distinct device (10DE:1DB5) that was missing entirely. Relabel 1DB1 as 16GB and add 1DB5. Restate the header comment as the actual generation rule and pin the corrected slugs (T4, A30, both V100 SXM2 parts) with a regression test that also asserts the tidy/short forms never reappear. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
da0dcf9edf |
test(platform): pin nil-safe GPU bundle wiring against null gpu values
The .Values.gpu accesses in the iaas bundle are fully parenthesized
((( .Values.gpu ).x).y) so the field walk stays nil-safe when an operator
nulls the whole gpu block or leaves a sub-key blank (an empty YAML key
parses to null, which Helm then drops from .Values). A bare dotted walk
would panic with "nil pointer evaluating interface {}.<field>" on the
first nil hop.
Add regression cases covering gpu: null in both passthrough and vgpu
variants plus blank sub-keys, and a comment warning against flattening
the parenthesized form. A mutation check confirms the cases fail with the
nil-pointer panic the moment the parentheses are removed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
91efa1f4eb |
feat(platform): auto-wire KubeVirt host-device support when gpu-operator is enabled
When cozystack.gpu-operator is in bundles.enabledPackages, the platform
now mirrors the chosen GPU variant into the KubeVirt CR — the HostDevices
feature gate is appended to developerConfiguration.featureGates, and a
starter permittedHostDevices block is filled from a per-variant defaults
table. Without this, a passthrough-variant GPU shows up under
.status.allocatable on the node but the KubeVirt admission webhook
rejects every VM that references it as a hostDevices.deviceName.
Two defaults tables ship under packages/core/platform/files/:
- gpu-passthrough-defaults.yaml covers Hopper (H100/H200), Ada
Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10),
Turing (T4), Volta (V100/V100S). PCI vendor:device pairs are
stable; resourceName slugs mirror nvidia-sandbox-device-plugin's
<arch>_<model>_<form>_<mem> convention. externalResourceProvider
is set on every entry because the resources are advertised by
the sandbox-device-plugin DaemonSet, not by KubeVirt itself.
- gpu-vgpu-defaults.yaml covers the Pascal-Ampere mdev profile
range the upstream vgpu-device-manager walks /sys/class/mdev_bus/
for. Ada Lovelace and Blackwell SR-IOV vGPU is out of scope for
the chart's default list (NVIDIA driver path differs) and is
expected to be advertised via the user-override hook below.
bundles.iaas.gpuOperatorVariant picks which table applies (default =
passthrough, vgpu = mediated). The same variant flows through to the
gpu-operator Package CR, replacing the previous always-default call.
Operators extend or replace the defaults via .gpu in platform values:
gpu:
replaceDefaults: false
permittedHostDevices:
pciHostDevices: [...]
mediatedDevices: [...]
mediatedDevicesConfiguration: {...}
replaceDefaults: false (the default) appends the user list to the
NVIDIA table; true drops the table entirely for clusters with only
non-NVIDIA accelerators or a strict allowlist requirement.
Companion helm-unittest suite covers the eight branches: no gpu
package emits no extras; gpu-operator at default variant emits the
HostDevices gate plus pciHostDevices; vgpu variant emits the
HostDevices gate plus mediatedDevices and mediatedDevicesConfiguration;
user overrides append by default; replaceDefaults wipes; unknown
variant fails fast.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
9352d3ee0e |
feat(gpu-operator): add container variant for preinstalled host driver
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.
The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.
- driver.enabled=false use the host driver
- toolkit.enabled=false host already has nvidia-container-toolkit;
enabling overwrites the containerd
config.toml the host configured via
nvidia-ctk runtime configure
- devicePlugin.enabled=true publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false do not unbind the host driver
- cdi.enabled=false CDI spec generation is serviced by the
disabled toolkit; an apt host uses the
legacy nvidia-container-runtime hook, so
pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false regular pod workloads, not VMs
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
80fbfa1ed8 |
fix(networking): point host ouroboros proxy at the root-tenant ingress (#2800)
## What this PR does With `publishing.proxyProtocol: true`, the host emits a `cozystack.ouroboros` Package using the no-override Package macro, so the proxy inherits the wrapper chart default backend `ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a managed Kubernetes tenant cluster, not the host: on the host, `extra/ingress` deploys ingress-nginx into the root tenant namespace as `root-ingress-controller` in `tenant-root`. The composed backend FQDN never resolves, so `ouroboros-proxy` never reaches Ready. This emits the host ouroboros Package with a `proxy.target` override derived from `publishing.ingressName` — namespace plus `<short>-ingress-controller` service name — mirroring the `trimPrefix "tenant-"` transform `extra/ingress` applies to its release namespace. The derivation reuses the existing platform invariant that `publishing.ingressName` is the namespace where the host ingress controller runs. Adds bundle helm-unittest coverage for the default ingress name and a non-default one, and corrects the misleading wrapper `values.yaml` comment that claimed the host uses `cozy-ingress-nginx`. Scope: this fixes the ouroboros readiness defect only. The separate defect where the injected nginx PROXY-protocol config keys are not consumed by the host ingress is tracked independently and not touched here. Closes #2797 ### Release note ```release-note fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced proxy protocol configuration to automatically route through the appropriate ingress-nginx service based on your ingress name settings. * **Tests** * Added test coverage for proxy protocol bundle wiring and ingress service routing. * **Documentation** * Clarified proxy target configuration behavior and fallback mechanisms. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
56014f1fdc |
fix(platform): order cozystack-basics after the APIs its admission policies type-check
cozystack-basics ships cluster-wide ValidatingAdmissionPolicies matching
tenants.apps.cozystack.io and the Gateway API types. KCM's VAP status
type-checker panics on a nil schema when it type-checks a policy against a
matched type that isn't resolvable yet; during a cold install this crashes
KCM, which stalls cert/token issuance, so cozystack-api never starts and the
tenants schema never resolves — a bootstrap deadlock.
Add dependsOn edges so the package installs only after cozystack-engine
(cozystack-api) and gateway-api-crds are Ready. Drift detection is off on
operator-generated HelmReleases, so a template capability gate would drop the
policy at first install and never re-add it; ordering is the reliable fix.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
2e23198b44 |
fix(platform): install velero by default
backupstrategy-controller is installed by default since |
||
|
Andrei KvapilandClaude
|
ad7cdece29 |
feat(platform): add migration 44 for the flux-tenants retirement
Defensively removes a leftover flux-tenants Deployment only when no HelmRelease carries sharding.fluxcd.io/key=tenants anymore (the flux-shard-operator backfill owns the live drain), and stamps cozystack-version 45 mirroring migration 43's labeled manifest. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
87ce7f6407 |
feat(fluxcd): retire the hand-rolled flux-tenants shard
The installer no longer ships the hand-rolled tenant helm-controller shard: flux-shard-operator provisions operator-managed shard Deployments cloned from flux-aio at runtime instead, and deletes a running flux-tenants Deployment only after the legacy tenants bucket drains. flux-aio itself stays unchanged and keeps owning everything without a shard key; the Makefile lines that synced the image/version into the removed manifest go away with it. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei KvapilandClaude
|
d75fac0096 |
feat(flux-shard-operator): add system package
New cozy-fluxcd system package (Deployment with 2 replicas, webhook cert via cert-manager, CREATE-only MutatingWebhookConfiguration with failurePolicy Ignore scoped to HelmReleases born on the legacy tenants bucket), registered in the platform system bundle and the root image build. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
f90411da7a |
fix(networking): point host ouroboros proxy at the root-tenant ingress
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros Package via the no-override Package macro, so the proxy inherits the wrapper chart default backend ingress-nginx-controller.cozy-ingress-nginx. That layout describes a managed Kubernetes tenant cluster; on the host, extra/ingress deploys ingress-nginx into the root tenant namespace as root-ingress-controller in tenant-root. The composed backend FQDN never resolves, so ouroboros-proxy never reaches Ready. Emit the host ouroboros Package with a proxy.target override derived from publishing.ingressName (namespace plus <short>-ingress-controller service name), mirroring the trimPrefix transform extra/ingress applies. Add bundle helm-unittest coverage for the default and a non-default ingress name, and correct the misleading wrapper values.yaml comment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
c435efa199 |
feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602)
## What this PR does Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and the embedded management-cluster manifests, and folds in the chart fixes that become hard errors under the new strict helm-controller v1.5. Flux v2.8's helm-controller v1.5.0 ships: - **Server-Side Apply with `--force-conflicts`** — strict CRD schema enforcement; misplaced fields (silently dropped on v2.7) now fail the apply. - **kstatus-based health checking** by default — parent HR waits for every applied resource (including child HRs) to be Ready before flipping its own Ready, surfacing latent ordering deadlocks. - **Helm v4 embedded** — `upgrade.force: true` is deprecated and now conflicts with SSA. - **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper readiness gating (used in PR #2601 split). Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same Flux upgrade triggers it. ## Commits **Flux upgrade itself:** - `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts to v0.48.0; web UI opt-in. - `feat(flux)`: regenerate embedded management-cluster manifests via `make update` in `packages/core/flux-aio` (timoni bundle build). **Chart fixes for strict SSA — fields the chart sent that v2.7 silently dropped, v2.8 rejects:** - `fix(kubevirt-instancetypes)`: drop persistent strip that produced null `preferredTPM` (folds #2612). - `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`, `minimumUptimeSecondsForBounce` from inside `automationOptions` to direct children of `spec`. - `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a phantom `template.spec`. - `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is set. - `fix(platform)`: drop deprecated `upgrade.force: true` from HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths. **Ordering / deadlock fixes under v2.8 kstatus:** - `fix(vpa)`: break circular wait between parent install and nested `vpa-for-vpa` HR. - `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant addon child HRs (parent waits on child via kstatus, child waited on parent — deadlock). **E2E waits for v2.8 kstatus timing:** - `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's faster dispatch). - `test(e2e)`: wait for parent HR Ready before downstream asserts in `run-kubernetes.sh` and `vminstance.bats`. ## Scope discipline This PR is part of the split of #2619 (the consolidated CI fixes branch) into review-friendly pieces. Companion PRs: - **PR #2601** (seaweedfs split) — folded into this PR (commits `29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption migration 43 (targetVersion 44), and the configurable db resources all land here, because the strict-SSA `upgrade.force` removal and the kstatus parent-HR timeout bump only make sense together with the split. #2601 is superseded. - **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) — independent, lands separately. - Several smaller standalone fixes lifted out of #2619 (startup probes, cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC test improvements) — opened as separate PRs. ## Verification - `helm template` renders cleanly for both `fluxcd` and `fluxcd-operator` packages with `web.enabled=false` (default) and `web.enabled=true`. - Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0 / v2.1.0 controller image strings. - No references to the v0.39-removed `--disable-wait-interruption` flag anywhere in `packages/` or `internal/`. ### Release note ```release-note Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking). When upgrading existing clusters: - Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too. - HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it. - KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass. - KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained. - FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Flux Status web UI: optional web server, config secret, service port, HTTPRoute/Ingress, network policy, and RBAC roles. * New SeaweedFS DB Helm chart and optional managed DB release. * **Improvements** * CRD/schema enhancements: new provider kinds, validations, variant option, and external checksum refs. * Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for reliability. * **Chores** * Bumped Flux Operator and Flux versions; documentation links updated to fluxoperator.dev. * **Bug Fixes** * Removed aggressive HelmRelease force-upgrade/install flags. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2d58479f06 |
feat(platform): add kuberture as optional system package (#2647)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional system package. Kuberture is a small controller that translates the `default/kubernetes` EndpointSlice into headless Services with external-dns-style annotations, closing the gap that external-dns cannot use EndpointSlice as a source directly: ```text EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS ``` The package is opt-in via `bundles.enabledPackages: [cozystack.kuberture]`. Operators populate `config.outputs` per HelmRelease — each output targets a specific external-dns instance by its `annotationPrefix`, so a single kuberture install can serve multiple external-dns instances (e.g. one publishing to the public zone, another to an internal corporate prefix) simultaneously. The vendored chart at `packages/system/kuberture/charts/kuberture/` is pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and pinned by both version AND OCI manifest digest in the Makefile (mirroring the ouroboros lockstep pattern). The controller image at `ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim values.yaml. Both pins advance in lockstep with each upstream release; the procedure is documented inline. Install-order dependencies declared in `PackageSource`: `cozystack.networking` (CNI before pod scheduling) and `cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the chart's enabled-by-default ServiceMonitor manifest). External-dns is deliberately NOT a dependency — kuberture creates annotated Services regardless of whether external-dns is installed; external-dns is a downstream consumer, not a runtime prerequisite. Coverage: - **helm-unittest** (`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases): fail-fast on empty outputs, ServiceMonitor enabled by default, source pinned to `default/kubernetes`, multi-output prefix isolation, empty-string `annotationPrefix` schema rejection, omitted-prefix fallback to controller default, ConfigMap → Deployment volume mount, controller image pinned by tag+sha256 digest, Role permits full Service lifecycle, metrics containerPort derived from `metricsBindAddress`. - **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package cozystack.kuberture` with two outputs (different `annotationPrefix` values), waits for HelmRelease and controller readiness, asserts both output Services are headless and carry the right `hostname`/`target`/`ttl` annotations under their configured prefix, cross-checks per-output isolation (each Service only carries its own prefix's annotations). ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not applicable — no UI changes. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Adds kuberture as an optional system package to publish the Kubernetes API endpoint to DNS via configurable external-dns outputs; includes Helm chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart values/schema. * **Tests** * Adds Helm unit tests and an end-to-end smoke test validating controller readiness, headless Service annotations, and external-dns routing/probe behavior. * **Documentation** * Adds a full README with usage, configuration examples, and vendoring/pinning guidance. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- Documentation companion: [cozystack/website#539](https://github.com/cozystack/website/pull/539). |
||
|
Myasnikov Daniil
|
910b505dc3 |
fix(fluxcd): sync app.kubernetes.io/version on the tenant flux Deployment
Address non-blocking review feedback from lexfrei: the flux-tenants sharded-controller Deployment carried a stale app.kubernetes.io/version: v2.7.3 while running helm-controller v1.5.0 (Flux 2.8.0). The flux-aio Makefile update target only synced the container image, so the version label drifted on every bump and make update never self-corrected. Bump the label to v2.8.0 and extend the yq patch to carry app.kubernetes.io/version from the source flux Deployment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
797284ab4f |
fix(platform): preserve no-delete label when migration 43 stamps version
Address review feedback from lexfrei on migrations/43: the version stamp did a label-less `kubectl create configmap | kubectl apply` under the same field manager migration 42 used, stripping the platform.cozystack.io/no-delete label and dropping cozystack-version out of the deletion-protection guardrail on every upgrade through this migration. Replace it with a labeled heredoc manifest mirroring migration 42 so the label survives the stamp. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f880b32495 |
fix(platform): bump migrations targetVersion to 44 so seaweedfs-db adoption runs
run-migrations.sh loops `seq $CURRENT $((TARGET-1))`, so with targetVersion: 43 a cluster at version 42 runs only migration 42 — migration 43 (the seaweedfs-db Cluster adoption introduced by the split) is never iterated, and the missing-file guard never fires because 43 is out of range. The migration is written as "43 -> 44" and stamps version=44. Without it the new seaweedfs-db HelmRelease hits a Helm ownership conflict on the existing Cluster and the seaweedfs-system upgrade can prune it (filer metadata loss). Set targetVersion to 44 so migration 43 runs, matching the migration-42 precedent. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
f17ead3d26 |
feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
29c6afc84d |
refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system
Splits the seaweedfs-system HelmRelease in two so the CNPG Cluster/seaweedfs-db lives in its own HR (seaweedfs-db), and the application HR (seaweedfs-system) dependsOn it. The new HR uses Flux v2 HelmRelease.spec.healthCheckExprs with a CEL expression on Cluster.status.conditions[type=Ready] plus waitStrategy.name: poller, so its Ready=True only flips after the postgres primary is actually serving connections — not just after helm install applied the Cluster CR. This eliminates the seaweedfs-filer CrashLoopBackOff race on a fresh tenant install. With Cilium kubeProxyReplacement: true, socket-LB returns EPERM from connect(2) to ClusterIPs with no Ready endpoints. Pre-split, the filer StatefulSet scheduled concurrently with the CNPG bootstrap (~55–70s of unavailable postgres), each connect() failed EPERM, kubelet exponential restart backoff pushed past the e2e bats 'kubectl wait hr/seaweedfs-system --timeout=2m' window, and the 'Configure Tenant and wait for applications' test failed. healthCheckExprs uses Flux's three-predicate form: route ClusterIsNotReady (the CNPG bootstrap condition) to `inProgress` so the HR keeps polling, keep `failed` for other Ready=False reasons, and rely on the HR `timeout: 10m` as the real backstop for a genuinely stuck cluster. A plain Ready=False predicate would flag the HR failed within seconds of creating the Cluster CR. Migration 42 adopts existing Cluster/seaweedfs-db resources into the new release on upgrade by rewriting meta.helm.sh/release-name and stamping helm.sh/resource-policy: keep so the seaweedfs-system upgrade (which no longer renders the Cluster) does not delete it during the transition. The application HR's dependsOn collapses the db dependency (unconditional) with the existing ingress dependency (guarded by `if eq $ingress .Release.Namespace` so sub-tenants that inherit ingress from a parent namespace don't deadlock on a non-existent local HR) into one list. Requires Flux v2.8.x (helm-controller v1.5.0+) for healthCheckExprs. Provided by the parent PR #2602 in this chain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
535f881160 |
fix(platform): include cert-manager in cozystack-engine oidc variant
The oidc variant installs system/cozystack-api and system/lineage-controller-webhook, both of which render cert-manager.io/v1 Certificate resources. Add cozystack.cert-manager to the oidc variant's dependsOn so it matches the default variant and webhook-cert consumers do not race the cert-manager controller on cold installs. Address review feedback from gemini-code-assist and coderabbitai on packages/core/platform/sources/cozystack-engine.yaml. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
69cb7b5f18 |
fix(platform): add cert-manager dependency to webhook-cert consumers
Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.
The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Timofei LarkinandClaude Opus 4.7
|
3bac5caac1 |
fix(platform): remove duplicate test target in Makefile
The deletion-protection commit added a second, identical 'test:' recipe to the platform Makefile, causing GNU Make to print 'overriding recipe' warnings on every 'make test' (and thus in helm-unit-tests CI output). Drop the duplicate; keep the original target. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Myasnikov Daniil
|
76fd5d1791 |
fix(platform): address lllamnyp review on PR #2650
unprotect.sh: add `gitrepository` to the namespaced-kind list so the
platform stays unprotectable when an operator configures
`sourceRef.kind: GitRepository`. Currently only `ocirepository` is
swept, while repository.yaml templates `kind: {{ $sourceRef.kind }}`
unrestricted.
run-migrations.sh: replace the silent skip on a missing migration file
with a hard error. All migrations 1..targetVersion exist in the shipped
image — a gap can only mean a packaging mistake (digest advanced
without the file landing), which previously stalled the cluster at a
stale version while reporting success.
migrations/42: drop the misleading "no 3-way-merge subtlety" claim from
the header. The apply records the no-delete label under the default
field manager's last-applied annotation, so a future label-less apply
by the same field manager would strip it. Document the constraint
instead of denying it.
hack/e2e-install-cozystack.bats: restore `helm.sh/resource-policy: keep`
on the reconstructed cozystack-version ConfigMap in the bypass-path
teardown so cluster state matches the chart template shape
(packages/core/platform/templates/cozystack-version.yaml) on exit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
2bed6eb090 |
fix(platform): do not swallow kubectl errors in unprotect.sh
The blanket `2>/dev/null || true` suppression masked RBAC, wrong-context, and API-unreachable failures, letting the script print "Done." while leaving labels in place. Replace with a helper that only tolerates the "unknown resource type" error (the actual best-effort case — kinds that don't exist on every cluster, e.g. linstorcluster on a non-LINSTOR install) and propagates everything else. Also keep stdout (resource list) and stderr (kubectl warnings) separate so deprecation warnings can't sneak into the loop variable. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
c2f1bb2eac |
feat(platform): make unprotect strips no-delete label for teardown
Address review feedback from @lexfrei on packages/core/platform/templates/ deletion-protection.yaml: with this PR every protected object — CRDs, ClusterIssuers, cozy-system and tenant-root Namespaces, cozystack-version ConfigMap, cozystack-packages Repository, tenant-root HelmRelease, LinstorCluster, plus the VAP and Binding themselves — carries the no-delete label. `helm uninstall cozy-platform` issues DELETE on each, and DELETE is denied for every one of them, so the chart effectively cannot be uninstalled without manual `kubectl label` against each object. Disaster recovery and dev-cluster reset both regress as a result. Add packages/core/platform/hack/unprotect.sh: enumerates the guarded set locally (per-kind, not via a single all-resources sweep — some kinds may not exist on every install, e.g. linstorcluster on a non-LINSTOR cluster, and kubectl errors on unknown kinds). For cluster-scoped kinds, lists objects with the label and labels them off one at a time. For namespaced kinds, lists (namespace, name) pairs via jsonpath and labels each in the right namespace. Idempotent: a label that is already absent is a no-op. Wire `make unprotect` in packages/core/platform/Makefile so the operator runs it before `helm uninstall`. The Makefile target carries a comment noting this is teardown / disaster-recovery only — after it runs, the cluster has no guardrail until the next `helm upgrade` re-stamps the label. Verified with a stub kubectl in PATH that the script issues the right get/label calls for both cluster-scoped (e.g. `kubectl label customresourcedefinition/packages.cozystack.io platform.cozystack.io/ no-delete-`) and namespaced kinds (e.g. `kubectl label --namespace tenant-root helmrelease tenant-root platform.cozystack.io/no-delete-`). The template-header reference at templates/deletion-protection.yaml that forward-links to `make unprotect` was added in the preceding L3 commit; the recipe it points at now exists. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
91aae36da3 |
feat(platform): NOTES.txt warns when no-delete VAP is silently skipped
Address review feedback from @lexfrei on packages/core/platform/templates/ deletion-protection.yaml: on a pre-1.30 cluster the capability gate (`.Capabilities.APIVersions.Has "admissionregistration.k8s.io/v1/ ValidatingAdmissionPolicy"`) evaluates to false and the policy + Binding render to nothing. There was no signal in `helm install`/`helm upgrade` output that the guardrail had been silently skipped, so operators on older clusters lost the protection without any way to discover that from the install log. Compounded the integration-test gap @lllamnyp called out in blocker 3. Add packages/core/platform/templates/NOTES.txt that renders a multi-line WARNING whenever the capability is absent: states the guardrail is NOT active, explains why (1.30+ required), notes that the label is still stamped on every protected object so the guardrail becomes active automatically on the next upgrade after a cluster bump, no migration step required. Add packages/core/platform/tests/notes_capability_gate_test.yaml: positive case asserts the warning text + the 1.30 hint + the label string render on a chart-level `capabilities.apiVersions: []`. Negative case asserts the warning is absent when admissionregistration.k8s.io/v1/ ValidatingAdmissionPolicy is in the capability list. Verified the positive case fails when the warning text is removed from NOTES.txt. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
919a8a9d05 |
docs(platform): clarify sequential self-protection on no-delete VAP
Address review feedback from @lexfrei on packages/core/platform/templates/ deletion-protection.yaml: both the ValidatingAdmissionPolicy and the ValidatingAdmissionPolicyBinding carry the no-delete label so each individually requires the bypass dance, but the labeling is symmetric, not truly bilateral. Removing the label from the Binding and deleting it first leaves the VAP labeled but unenforced, so the VAP itself is then deletable without the bypass — and vice versa. That is by design: the guardrail is operational guidance, not adversarial defense against an operator with cluster-admin who is determined to remove it. Add a paragraph in the template comment header so a future reader does not waste time wondering why the symmetric self-protection isn't truly bilateral. Forward-references the whole-chart teardown recipe (make unprotect) so the comment links to the documented escape hatch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e34eeb0ee7 |
fix(platform): document --namespace bypass and add VAP unit tests
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml:
The bypass instructions in both the template-author-facing comment and the
user-facing CEL message did `kubectl label <kind> <name>` with no
--namespace flag. Several protected objects are namespaced (cozystack-version
ConfigMap, cozystack-packages Repository, tenant-root HelmRelease), so an
operator who copies the documented command verbatim against their default
context gets a "not found" error and concludes that the bypass mechanism is
broken. Updated both call sites to show --namespace <ns> with an explicit
"(omit for cluster-scoped resources)" caveat.
The reviewer also flagged that the VAP contract is otherwise unasserted by
unit tests. Add packages/core/platform/tests/deletion_protection_test.yaml
covering:
- both the VAP and the Binding carry platform.cozystack.io/no-delete=true
on themselves (the self-protection contract)
- failurePolicy is intentionally absent on the VAP
- the deny message contains the bypass command WITH --namespace and the
cluster-scoped caveat (regression catch for this fix; verified failing
when --namespace is dropped from the template)
- the Binding's objectSelector + validationActions: [Deny] still match
the no-delete label
Wire packages/core/platform/Makefile's test target to helm unittest so
hack/helm-unit-tests.sh actually runs the new suite in CI (the existing
tests/ files were orphaned without it).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
ecdf3f3a15 |
fix(platform): drop redundant failurePolicy on no-delete VAP
Address review feedback from @lllamnyp on packages/core/platform/templates/ deletion-protection.yaml: the VAP's validation expression is the literal `false` — no parameters, no variables, no references — so CEL evaluation cannot fail and the api-server never has occasion to apply failurePolicy. The previous `failurePolicy: Fail` line was a no-op masquerading as operational config. Drop the field (the default is Fail anyway in admissionregistration.k8s.io/v1) and document in the template comment why it is intentionally absent, so the next reader does not re-add it on the assumption that it was forgotten. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
052f070b63 |
fix(platform): migration 42 uses a single declarative apply
Address review feedback from @lllamnyp on packages/core/platform/images/ migrations/migrations/42: the previous version did `kubectl label --overwrite` then `kubectl apply` on a manifest with no labels, relying on `kubectl apply`'s 3-way strategic merge to leave the imperatively-set label intact. That works in the steady state but is fragile — the contract is "the label is preserved because it is never in last-applied-configuration" — and it is inconsistent with the helm template at templates/cozystack-version.yaml, which bakes the label inline. Replace the two-step dance with one declarative apply that includes the deletion-protection label, matching the chart template. One source of truth, no last-applied-merge subtlety. The script remains idempotent (apply is a no-op once the ConfigMap matches). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
c7227bc666 |
feat(platform): migration 42 backfills no-delete label on cozystack-version
Address review feedback from coderabbitai on
packages/core/platform/templates/cozystack-version.yaml:10:
existing clusters skipped the label because the template's
`lookup`+`if not $configMap` guard never re-renders the ConfigMap on
upgrade (the skip is intentional — it prevents helm from clobbering the
data.version that the migration controller writes from its pre-upgrade
hook).
A one-shot migration is the correct mechanism here:
- Migration 42 in packages/core/platform/images/migrations/migrations/42
runs `kubectl label --overwrite configmap cozystack-version
platform.cozystack.io/no-delete=true` and stamps version=43. Slot 42
rather than 40 because main already occupies 40 (dashboard CRD
cleanup) and 41 (kubernetes ephemeralStorage→diskSize rename).
- Bump migrations.targetVersion 42 -> 43 so the hook fires on the
next platform upgrade.
Idempotent: --overwrite is a no-op when the label is already at the
target value, MIGRATION_DRY_RUN=1 prints without applying, and the
script aborts before stamping if the kubectl label call fails.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
e5dd5681c3 |
feat(platform): self-protect the deletion-protection VAP and Binding
Address review feedback from gemini-code-assist on packages/core/platform/templates/deletion-protection.yaml:15: label the ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding themselves with platform.cozystack.io/no-delete=true so the guardrail guards itself. A direct kubectl delete on either resource now returns the same denial as deletion of any other protected object; the same two-step label-then-delete bypass applies when intentional removal is needed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
43ef456d84 |
feat(platform): gate deletion-protection VAP on Kubernetes 1.30+ capability
Address review feedback from gemini-code-assist on packages/core/platform/templates/deletion-protection.yaml:42: wrap the ValidatingAdmissionPolicy and Binding in a Helm capability check so the template stays renderable on clusters that pre-date 1.30 (when ValidatingAdmissionPolicy graduated to GA). The check targets the specific Kind, not the API group/version — admissionregistration.k8s.io/v1 itself has existed since 1.16 for ValidatingWebhookConfiguration, so the group/version alone is not a meaningful capability gate. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
