mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
99a28d82c21cbb4fd341db9a9503d2830a7600a1
4851
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 myasnikovdaniil
|
99a28d82c2 |
test(e2e): migrate app suite from BATS to Chainsaw and wire into CI (#2826)
## What this PR does Migrates the E2E **app** test suite from BATS to [Kyverno Chainsaw](https://github.com/kyverno/chainsaw) — a declarative, Kubernetes-native E2E framework (CNCF, part of the Kyverno project; successor to KUTTL) — and wires it into CI as the replacement for the per-app BATS loop. This grows the original pilot (postgres + bucket) into the full suite. All 22 app suites are ported under `hack/e2e-chainsaw/` and the `hack/e2e-apps/*.bats` files are removed: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`, `clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`. **Approach** - **Declarative** suites assert on `status.conditions` and concrete fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait` pair that appeared ~100 times collapses into a single `assert` that polls existence and state together, with a structured diff on failure and automatic `events`/`describe`/`podLogs` capture via `catch` (previously only `harbor.bats` did this, by hand). - **Imperative** suites keep their logic in `script` steps: `openbao` init/unseal, `kuberture` external-dns split-horizon probes, `vminstance`, the `gateway` admission/impersonation cases, and `kubernetes-latest`/`previous`, which wrap the relocated `hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up, LB/NFS/ouroboros checks). - `gateway` tests derive the tenant apex from the namespace `namespace.cozystack.io/host` label at runtime, so they are host-independent. **CI** - The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new `test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`. - The `chainsaw` binary is added to the e2e-sandbox image. - `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap + OpenAPI checks). - The per-app 3-retry loop is dropped — assertion polling replaces the fixed-timeout flakiness it papered over. **Validation** Ran against a development cluster: the DB/app, storage, and VM suites pass; the three suites that depend on platform features not present on that cluster (`gateway`, `kuberture`, `external-dns`) and the two heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a freshly installed platform. Note for reviewers: service-port asserts use the `(ports[*].port)` projection form rather than a number-literal filter (`` ports[?port == `N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal comparisons. **2026-07-09: reconciled with main (~520 commits of drift)** - etcd suite re-ported to the v1alpha2 operator contracts (#2859): `readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip driving `examples/backups/etcd` - 12 new `ingress-hostname-policy` gateway cases ported (apex derived at runtime from the tenant-root namespace label) - post-branch bats drift folded in: `mariadb-single` webhook guard, kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant PVC-reclaim guard (#3059), kuberture fail-loud negations, SC-fallback-default test (#2872 B1) - 4 suites that only exist on main since the branch was cut are ported: `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure` - CI seams: `nightly.yaml` converted to the chainsaw invocation, `run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh` replicated in the Chainsaw global catch ### Release note ```release-note fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * End-to-end testing is now driven by a unified Chainsaw suite (platform apps, Kubernetes, networking, security groups, service exposure). * Test Impact Analysis selects affected suites, with an option to run the full suite. * CI produces a JUnit report artifact and collects suite-scoped diagnostics on failure. * **Bug Fixes** * CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and sandbox bucket state diagnostics. * Improved teardown reliability with bounded wait + finalizer handling to avoid hangs. * **Documentation** * Updated E2E testing docs and local guidance to reflect the new Chainsaw-based workflow and suite paths. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
8af4ed543e |
test(e2e): address review nits — shellcheck directive + ports-workaround TODO
Non-blocking review notes from #2826: - Add `# shellcheck shell=bash` to the four sourced `_lib/*.sh` helpers (etcd-cleanup, remediation-guard, run-kubernetes, talos-image-cache) so shellcheck stops flagging SC2148 (no shell directive) — they are only ever `. sourced`, so there is no runtime interpreter to infer from a shebang. - Add a `# TODO(chainsaw>0.2.15)` marker at the postgres Service asserts explaining the `(ports[0].port)`/`(ports[*].port)` projection is a v0.2.15 number-literal-filter workaround (and that `ports[0]` checks only the first port), so it gets reverted on a chainsaw bump; cross-references the same workaround in kafka/mariadb/harbor and the e2e-testing.md gotchas section. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
18715f8c63 |
docs(agents): document Chainsaw v0.2.15 gotchas in the e2e-testing guide
The v0.2.15 syntax/semantics traps were tribal knowledge scattered across PR review notes (the number-literal caveat lived only in #2826's body, which evaporates on merge). Consolidate them into a durable subsection so the next author finds them before hitting each one fresh: filter-as-list conditions, no number-literals/backticks in JMESPath, error: fails hard on an unknown GVK, no conditional Test-level skip, step-level-only finally, bounded/blocking cleanup surfacing latent teardown bugs, projection-style failure diffs, and the catch-script cwd. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
b99df11b36 |
fix(kubernetes): bump tenant worker Talos to v1.13.6 (#3269)
## What this PR does Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24. That kernel predates the fixes for two related KVM x86 shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`: CVE-2026-46113, fixed upstream in `0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and released in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables nested virtualization inside the VM, and either bug lets a nested guest escape into the worker. This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38. Stable kernel branches are cumulative, so v1.13.6 is the first v1.13 patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113. Nested virtualization stays available to tenants; the kernel bump is the fix, not a capability removal. The image-factory schematic is unchanged and resolves for the new version: both the `openstack-amd64.raw.xz` artifact and the `installer/<schematicID>:v1.13.6` manifest are served by the public factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the chart's Talos-to-Kubernetes support window is unaffected and no tenant `spec.version` needs to move. Applying this rolls each worker pool once. The new pin changes the rendered worker spec, which rotates the content hash in the `KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI replaces the worker VMs to deliver the new kernel. The pinned hashes in the chart tests are recomputed to match. A second commit fixes the support-matrix lookup itself. The matrix dict was keyed on the full patch version, so every Talos patch bump dropped out of the dict and silently disabled the render-time guard — exactly the mismatch the guard exists to catch. The lookup now derives the minor from `talos.version`. The supported-Kubernetes window is unchanged; unknown minors are still skipped, leaving compatibility to whoever overrode the pin. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated the default Talos worker release to v1.13.6 for Kubernetes cluster provisioning. - Improved compatibility checks to apply consistently across Talos patch releases within the same minor version. - **Bug Fixes** - Corrected generated resource naming expectations when storage settings or GPU configuration change. - **Tests** - Added coverage confirming Kubernetes clusters render successfully with Talos v1.13.6. <!-- end of auto-generated comment: release notes by coderabbit.ai -->v1.6.0-rc.2 |
||
|
myasnikovdaniil
|
c8ee5dc317 |
chore(deps): update container image & action digests (#3232)
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [actions/stale](https://redirect.github.com/actions/stale) ([changelog](https://redirect.github.com/actions/stale/compare/eb5cf3af3ac0a1aa4c9c45633dd1ae542a27a899..1e223db275d687790206a7acac4d1a11bd6fe629)) | action | digest | `eb5cf3a` → `1e223db` | | alpine | final | pinDigest | → `28bd5fe` | | alpine | stage | pinDigest | → `28bd5fe` | | [altinity/clickhouse-backup](https://redirect.github.com/Altinity/clickhouse-backup) | final | pinDigest | → `b8a098d` | | clickhouse/clickhouse-server | final | pinDigest | → `f21e0f9` | | debian | final | pinDigest | → `ee12ffb` | | debian | final | pinDigest | → `30482e8` | | debian | stage | pinDigest | → `30482e8` | | debian | final | pinDigest | → `28de087` | | docker.io/golang | stage | pinDigest | → `079e598` | | docker.io/library/alpine | final | pinDigest | → `1435830` | | docker.io/library/alpine | final | pinDigest | → `28bd5fe` | | docker.io/library/golang | stage | pinDigest | → `079e598` | | [docker/dockerfile](https://redirect.github.com/moby/buildkit) | syntax | pinDigest | → `e2a8561` | | gcr.io/distroless/static | final | pinDigest | → `d29e660` | | gcr.io/distroless/static | stage | pinDigest | → `d29e660` | | golang | final | pinDigest | → `0178a64` | | golang | stage | pinDigest | → `079e598` | | golang | stage | pinDigest | → `0178a64` | | golang | final | pinDigest | → `079e598` | | [grafana/grafana](https://redirect.github.com/grafana/grafana) | final | pinDigest | → `f739273` | | [nginxinc/nginx-unprivileged](https://redirect.github.com/nginx/docker-nginx-unprivileged) | final | pinDigest | → `0c79d56` | | [node](https://redirect.github.com/nodejs/node) | stage | pinDigest | → `16e22a5` | | quay.io/centos/centos | final | pinDigest | → `0c9db82` | | quay.io/cilium/cilium | stage | pinDigest | → `20fbbc1` | | [quay.io/poseidon/matchbox](https://redirect.github.com/poseidon/matchbox) | final | pinDigest | → `06bcdae` | | ubuntu | final | pinDigest | → `4fbb8e6` | | ubuntu | final | pinDigest | → `0e0a0fc` | | ubuntu | stage | pinDigest | → `0e0a0fc` | --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - "before 6am on monday" - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/cozystack/cozystack). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==--> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
302a72f14a |
Merge origin/main into feat/e2e-chainsaw-example
Reconciles the talos-image-cache conflicts created when #3254 merged to main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh, a superset of #3254's logic + the node-join diagnose helper), drop the BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path versions of the mirror manifest, install bats, and unit test. Also picks up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and Talos v1.13.6 (#3240). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
02b87b63f2 |
fix(apps/kubernetes): bound the pre-delete hook's tenant HelmRelease wait (#3271)
The pre-delete hook (delete.yaml) deleted the tenant's addon HelmReleases with
an unbounded `kubectl delete … --wait=true`. When the tenant has no working
nodes (LINSTOR/ZFS exhaustion, worker VMs that never joined, or a nodeGroup at
minReplicas: 0), those uninstalls cannot converge — in-tenant cilium stays
InstallFailed because cilium-operator has nothing to schedule on — so the hook
hung indefinitely, the parent Helm uninstall failed on the pre-cleanup Job, and
the Kubernetes CR was stuck deleting.
Bound the wait with --timeout=120s; on expiry, force-clear the Flux finalizer
from any child HelmRelease still present so it can be garbage-collected — safe
because the whole tenant (control plane torn down in the steps below) is being
destroyed, so there is nothing left to orphan. A healthy tenant's children
uninstall well within the timeout and never reach the force path.
Also unblocks the render-side kubernetes-oidc-{system,customconfig} Chainsaw
e2e suites, which deterministically hit this teardown hang under the Chainsaw
cleanup budget.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
renovate[bot]
|
5b9b420ffe | chore(deps): update container image & action digests | ||
|
Aleksei Sviridkin
|
a3829ab66c |
fix(talos): bump Talos to v1.13.6 (CVE-2026-53359) (#3240)
## What this PR does Bumps the platform Talos profiles from v1.13.5 to v1.13.6. CVE-2026-53359 and CVE-2026-46113 are related KVM x86 shadow-paging use-after-free bugs that let a guest escape to the host, and the fix is complete only once both upstream patches (`81ccda30b4e8`, `0cb2af2ea66a`) are applied. On the 6.18 stable line CVE-2026-46113 landed in 6.18.30 and CVE-2026-53359 in 6.18.38. Talos v1.13.5 runs Linux 6.18.36, so it carries only the first of the two. Talos v1.13.6 runs Linux 6.18.38 and carries both. Nested virtualization stays enabled. The bugs are fixed in the kernel, so there is no reason to take the feature away from workloads that rely on it — bumping the kernel is the complete fix. Profiles are regenerated with `hack/gen-profiles.sh v1.13.6`. Alongside the Talos version and the installer reference, this refreshes the pinned system-extension digests: the firmware extensions move from the 20260519 to the 20260622 build, and `intel-ucode` keeps its `20260512` tag but resolves to a new digest because upstream re-pushed that tag as an OCI image index (identical config and layer blobs underneath). References: [Talos v1.13.6 release notes](https://github.com/siderolabs/talos/releases/tag/v1.13.6) (`Linux: 6.18.38`) ### Screenshots Not applicable, no UI changes. ### Release note ```release-note fix(talos): bump Talos Linux to v1.13.6, which ships Linux 6.18.38 with the kernel fixes for the CVE-2026-53359 and CVE-2026-46113 KVM guest-to-host escapes ``` |
||
|
myasnikovdaniil
|
06b17bf7a9 |
ci: move remaining workflows off the persistent self-hosted runner (#3268)
## What this PR does Completes the last step of #2937 — no workflow depends on the persistent self-hosted runner anymore, so it can be decommissioned. PR builds, the main cache warmer, nightly, promote-rc, and pull-requests-release already run on the ephemeral pool; this PR moves the stragglers: - **`tags.yaml` / `prepare-release`** moves to the large ephemeral shape (same as the main cache warmer). Release builds now warm-start from the shared `mode=max` build cache in OCIR (written by `build-main.yaml` on every push to main) while still pushing images to GHCR. Cache manifests are registry-portable, and a missing cache ref degrades harmlessly into a cold build. Without this wiring, every tag build on an ephemeral runner would start 100% cold — the persistent runner used to warm-start from local buildkit state. - **`tags.yaml` / `generate-changelog` and `update-website-docs`**, **`backport.yaml`**, **`update-releasenotes.yaml`** move to GitHub-hosted runners: these jobs only need git and the GitHub API, no docker or repo toolchain. - **`pull-requests.yaml`**: the `debug` label no longer reroutes jobs to the self-hosted runner — that path would queue forever once the runner is gone. The label still gates the SSH breakpoint on e2e failure, which works from ephemeral runners (the breakpoint connects out to the rendezvous server). - **`flux-shard-operator`** was the last image built with the stale inline/`:latest` cache pattern (always cold since #2711); it now uses the shared `cache-args` registry cache like every other image. Operational notes for the actual decommission: - The breakpoint rendezvous server (`BREAKPOINT_ENDPOINT`) is separate infrastructure — if it happens to live on the same host as the runner, it needs a new home before the host is retired. - `tags.yaml` now uses the `OCIR_USER`/`OCIR_TOKEN` secrets on tag events (already used there by `nightly.yaml` and `release-e2e` before it). ### Screenshots Not applicable — CI-only change. ### Release note ```release-note ci: all CI jobs now run on ephemeral or GitHub-hosted runners; release tag builds warm-start from the shared registry build cache ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated multiple GitHub Actions workflows to use more consistent hosted runner environments for checks, PR validation, tagging, backports, and release note updates. * Improved tag/release build setup for ephemeral runners, including optional build toolchain setup and container registry authentication. * Enhanced image build caching for the shard operator to improve packaging efficiency. * **Bug Fixes** * Removed label-driven runner switching in pull request checks to make build and test execution more predictable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
814a47f23e |
ci: move remaining workflows off the persistent self-hosted runner
Finishes the runner decommission tracked in #2937. PR builds, the main cache warmer, nightly, promote-rc and pull-requests-release already run on the ephemeral pool; this moves the stragglers so no workflow depends on the persistent runner anymore: - tags.yaml prepare-release: large ephemeral shape (same as the cache warmer), flux CLI bootstrap, and read-only OCIR access so release builds warm-start from the shared mode=max build cache while still pushing images to GHCR. Without the cache wiring every tag build on an ephemeral runner would be 100% cold. - tags.yaml generate-changelog / update-website-docs, backport.yaml, update-releasenotes.yaml: GitHub-hosted runners — these jobs only need git and the GitHub API, no docker or repo toolchain. - pull-requests.yaml: the debug label no longer reroutes jobs to the self-hosted runner (that path would queue forever once the runner is gone). The label keeps gating the SSH breakpoint on e2e failure, which works from ephemeral runners too. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
681f4a8313 |
build(flux-shard-operator): read the shared registry build cache
flux-shard-operator was the last image built with the pre-#2937 cache pattern: --cache-from a :latest tag that PR builds no longer publish and --cache-to type=inline, which never caches multistage builder layers. Use the shared cache-args macro so it reads (and, on main, writes) the same mode=max :buildcache ref in CACHE_REGISTRY as every other image. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
3e9b5600e3 |
fix(talos): bump Talos to v1.13.6 (CVE-2026-53359)
Talos v1.13.5 ships Linux 6.18.36, which carries the fix for CVE-2026-46113 but not the one for CVE-2026-53359. The two are related KVM x86 shadow-paging bugs allowing a guest to escape to the host, and both patches are needed for the mitigation to hold. Talos v1.13.6 ships Linux 6.18.38, the first release on the 6.18 stable line that has both. Profiles regenerated with hack/gen-profiles.sh v1.13.6. Alongside the Talos version and the installer reference, this refreshes the pinned system-extension digests: the firmware extensions move from the 20260519 to the 20260622 build, and intel-ucode stays on its 20260512 tag but resolves to a new digest because upstream re-pushed that tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
IvanHunters
|
ca93adc728 |
test(e2e): make the in-sandbox Talos image cache reachable by tenant workers (#3254)
## What this PR does The in-sandbox Talos image cache added in #3244 is unreachable by the tenant workers it was meant to serve, so it makes the `kubernetes-latest` / `kubernetes-previous` e2e tests fail deterministically instead of relieving the intermittent public-factory flake it targets (#3231). Every branch off current `main` inherits this — for example #3240 has failed its tenant-Kubernetes e2e repeatedly for this reason alone, despite being unrelated (it only touches host boot assets). Root cause: tenant namespaces run under a default-deny Cilium egress installed by the tenant chart. A worker's CDI importer is allowed to reach the outside world (so the public factory works) and kube-dns (so the name resolves), but nothing permits egress to an arbitrary `kube-system` Service. The importer therefore resolves `talos-image-cache.kube-system.svc` yet its TCP connect to the ClusterIP is silently dropped (`dial tcp <clusterip>:80: i/o timeout`), the DataVolume import never starts, the worker VM stays `DataVolumeError`, and no tenant node joins within the 12-minute deadline. #3244's readiness gate never caught this because it probed the server over `localhost` inside the serve Pod, which passes regardless of cross-Pod reachability. Two changes: - Ship a tightly-scoped `CiliumClusterwideNetworkPolicy` that lets importer Pods in `tenant-test` egress to the mirror Pod in `kube-system`. Cilium unions allow rules, so this only adds a hole; scoping it to a namespace that is already default-deny egress means it never flips another namespace's posture, and the mirror Pod needs no ingress rule because no ingress policy selects it. The policy is applied at point-of-use (not with the rest of the mirror manifest, which is applied before Cilium's CRDs exist). - Replace the localhost range-probe with one that faces the exact same network path as a real importer: a throwaway Pod in the tenant namespace, labelled `cdi.kubevirt.io=importer`, that fetches the seeded image from the Service ClusterIP with a Range request and must get a `206`. Tenants are pointed at the mirror only when that end-to-end check passes; otherwise the harness falls back to the public factory. This restores the invariant #3244 intended — the mirror can only help, never make CI worse. Not a retry/timeout bump: it fixes the real network path and adds a genuine reachability gate. ### Screenshots N/A — CI-only change, no user-facing surface. ### Release note ```release-note test(e2e): make the in-sandbox Talos image cache reachable by tenant Kubernetes workers, fixing deterministic kubernetes-* e2e failures (#3231) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Updated Talos image cache e2e selection to run a tenant-scoped reachability probe to the local mirror and require HTTP byte-range support (`206`) before using it. * If the probe fails or does not confirm byte-range behavior, the flow now reliably falls back to the public Talos image source. * Added CI-only tenant egress controls to allow importer Pods to reach the mirror Service. * **Documentation** * Expanded e2e guidance to clarify the mirror is best-effort and that fallback happens at probe time. * **Tests** * Added a new Bats suite covering manifest splitting, policy wiring, probe overrides, and strict `206` gating/fallback scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
6b1e170e44 |
feat(release): add Cut Pre-release Tag workflow (#2677) (#3253)
## What this PR does Adds `cut-prerelease.yaml` — a `workflow_dispatch` that becomes the **sole entry point for creating a pre-release tag** (`vX.Y.Z-alpha.N` / `-beta.N` / `-rc.N`), replacing the manual `git push origin HEAD:refs/tags/<tag>`. ## Why #2677 makes stable tags immutable, but tag *creation* was still only convention-enforced for **pre-releases**. Repo rulesets **can** already restrict *stable* `vX.Y.Z` creation to CI — a "Restrict creations" ruleset targeting `v*` with an exclude pattern for `v*-*` covers stable tags without any new workflow. What that approach **cannot** cover is pre-releases: `-rc` / `-alpha` / `-beta` tags were hand-pushed, so they can't be locked to CI while they still have a human entry point. This workflow closes that gap by making CI the sole creator of pre-release tags too. With it in place, an admin can extend "Restrict creations" (CI app as sole bypass) to **all** of `v*` — pre-release and stable alike — instead of only the stable subset. _(Correction: an earlier version of this description claimed a ruleset couldn't express "stable-only" creation. It can, via include + exclude patterns — thanks @IvanHunters. The workflow's real value is extending creation-control to pre-release tags, plus the audit trail, single entry point, guaranteed `base_ref`, and app-identity push that makes `tags.yaml` fire.)_ ## How it works - Generates the CozyStack CI app token (same app as `promote-rc.yaml`) and pushes the tag **as the app** — so `tags.yaml` fires (a tag pushed with the default `GITHUB_TOKEN` would not) and the push event's `base_ref` is populated (`tags.yaml`'s *Get base branch* step requires it). - Validates `^v\d+\.\d+\.\d+-(alpha|beta|rc)\.\d+$` and **rejects stable `vX.Y.Z`** — stable tags are cut write-once by the promote flow at a PR merge commit, never here. - Gates on the dispatcher having **write access** to the repo (`workflow_dispatch` already requires it; the explicit check is auditable defense-in-depth that fails closed). - Enforces dispatch from `main` (new minor's `vX.Y.0-*` only) or the matching `release-X.Y` branch; write-once (refuses an existing tag) with a stale-tip guard so a branch that advanced since dispatch fails cleanly instead of burning the tag name. - Pushes `HEAD:refs/tags/<tag>` from the branch tip — the exact form `docs/release.md` prescribes. Runs on `ubuntu-latest` (no build runner needed). Also updates `tags.yaml`'s two human-facing error messages and `docs/release.md` (the RC and patch flows, the workflow list, and a new repo-side tag-ruleset enforcement note) to point at the workflow instead of a manual push. Completes SHA-pinning on the touched release workflows (`nightly.yaml`, `promote-rc.yaml`, `retention.yaml`) — the zizmor "Audit workflows" gate was red on `main` for those. ## Rollout ordering The two `v*` tag rulesets have a hard order (now documented in `docs/release.md`): 1. **Immutability** (*Restrict updates* + *deletions*, no bypass) — safe to enable at any time; nothing in the flow ever moves or deletes a `v*` tag. 2. **Creation control** (*Restrict creations*, CI app sole bypass) — enable **only after this PR ships**, since this workflow is what makes CI the sole creator of pre-release tags. Enabling it earlier would lock maintainers out of cutting rc/alpha/beta. ## Validation - `actionlint` — clean. - `act -l` — job graph resolves (`cut` / `workflow_dispatch`). - `zizmor --offline` — no findings across the whole workflows dir (SHA-pinned actions, least-privilege `contents: read`). - Run blocks `bash -n` syntax-checked. - **Untested locally:** the workflow runtime (app-token auth, the real tag push, `base_ref` population, the `getCollaboratorPermissionLevel` gate) needs a live run with the CI secrets — the same "orchestration is unproven until it runs" caveat #3017 carried. ## Review follow-up Addressed the non-blocking findings from @IvanHunters' review (commit `fix(ci): address review findings`): 1. **[MINOR]** `main` dispatch now requires patch `Z == 0` (a new minor's pre-release), so a patch-line rc can't be cut from `main` content. 2. **[MINOR]** Added a stale-tip guard: if the dispatch branch advanced between dispatch and push, fail with a re-run hint instead of tagging a stale commit / burning the write-once name on an empty-`base_ref` failure. 3. **[MINOR]** Corrected the PR-body rationale above (rulesets can express stable-only creation via include + exclude). 4. **[NIT]** `ls-remote` now distinguishes "tag absent" (exit 2) from a transport/auth failure. 5. **[NIT]** Shallow checkout (`fetch-depth: 1`, no `fetch-tags`); dropped the redundant local tag pre-check. 6. **[NIT]** Pure-input validation now runs before the token mint + checkout (fail-fast). Plus a **write-access gate** on the dispatcher, requested separately. Part of #2677. ## Release note ```release-note NONE ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a controlled workflow to create pre-release tags (alpha/beta/rc) with write-once behavior. * Added stronger enforcement so pre-releases are cut from the correct release line. * **Bug Fixes** * Improved guidance when stable tags are rejected or when base branch info is missing. * Added safeguards to prevent cutting tags from an out-of-date branch tip. * **Documentation** * Updated release process docs, including new tag protection rulesets for `v*` tags. * **Chores** * Pinned multiple CI workflow actions to fixed versions for reliability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
ff0d0d77a7 |
test(e2e): make the talos image cache reachable by tenant workers (#3254)
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant workers it serves, so the kubernetes-* e2e tests fail deterministically at the 12m node-join wait instead of relieving the public-factory flake (#3231). Root cause (from #3254 by @lexfrei, verified against the failing run's crust-gather snapshot): tenant namespaces run a default-deny Cilium egress (allow-internal/external-communication + the <tenant>-egress CCNP) that permits the world entity and the tenant tree but NOT an arbitrary kube-system Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc yet its TCP connect to the ClusterIP is silently dropped and no disk imports. #3244's gate never caught this because it probed the server over localhost inside the serve pod, which passes regardless of cross-pod reachability. Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/): - ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in tenant-test -> mirror pod in kube-system), applied at point-of-use once Cilium CRDs exist; excluded from the pre-Cilium install apply via yq. - replace the readiness probe with a throwaway pod in tenant-test labelled cdi.kubevirt.io=importer that curls the ClusterIP with a range request and must get 206 - the exact egress path a real importer faces; tenants use the mirror only when that end-to-end check passes, else fall back to the public factory, so it can only help, never make CI worse. - unit test pinning the manifest split, probe/policy label agreement, the strict 206 gate, and the --overrides JSON builder. Also adds (a2) DataVolume import-stage diagnostics to the node-join failure block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via talos_image_cache_diagnose) so the import-stage failure mode is legible. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
b73860d4c9 |
fix(kubernetes): bump tenant worker Talos to v1.13.6
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24. That kernel predates both fixes for a pair of related KVM x86 shadow-paging use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359, fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables nested virtualization inside the VM, and either bug lets a nested guest escape into the worker. Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0 ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries only CVE-2026-46113. Nested virtualization stays available to tenants — the kernel bump is the fix. The new pin changes the rendered worker spec, so the content hash in the KubevirtMachineTemplate and talos-reconcile Job names rotates and each worker pool rolls once on upgrade. Pinned hashes in the chart tests are recomputed accordingly. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
1a0680b778 |
test(e2e): allow tenant CDI importers to reach the image cache
Tenant namespaces run under a default-deny Cilium egress installed by the tenant chart, which lets a worker's CDI importer reach the outside world and kube-dns but nothing in kube-system. The in-sandbox image mirror is a kube-system Service, so without an explicit allow the importer's connect to its ClusterIP is dropped and the worker disk import never starts. Ship a tightly-scoped CiliumClusterwideNetworkPolicy in the mirror manifest that lets importer Pods in tenant-test egress to the mirror Pod in kube-system. Cilium unions allow rules, so this only adds a hole; scoping it to tenant-test (already under default-deny egress) means it never flips another namespace's posture, and the mirror Pod needs no ingress rule because it is selected by no ingress policy. The policy is kept out of the pre-install apply because that runs before Cilium's CRDs exist; the resolver in hack/e2e-apps/talos-image-cache.sh applies it at point-of-use once Cilium is up. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f5ec885d8d |
test(e2e): gate the talos image cache on real tenant reachability
The in-sandbox Talos image mirror was pointed at by tenant Kubernetes CRs whenever its Deployment was Available, but "Available" only proves the seed finished and the server answers on localhost. It does not prove a worker's CDI importer can reach the Service. Tenant namespaces run under a default-deny Cilium egress that permits the outside world and kube-dns but not an arbitrary kube-system Service, so the importer resolves the name yet its TCP connect to the ClusterIP is silently dropped. The disk import then never starts, no tenant node joins within the 12-minute deadline, and the kubernetes-* tests fail deterministically -- worse than the intermittent public-factory flake the mirror was meant to relieve. Replace the localhost range-probe with one that faces the exact same network path as a worker importer: a throwaway Pod in the tenant namespace, labelled cdi.kubevirt.io=importer, that fetches the seeded image from the Service ClusterIP with a Range request and expects a 206. Point tenants at the mirror only when that end-to-end check passes; otherwise fall back to the public factory, restoring the guarantee that the mirror can only help, never make CI worse. The resolver also installs the importer egress allow at point-of-use, because Cilium's CRDs do not exist yet when the mirror manifest is first applied before install. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b420d3de02 |
fix(kubernetes): key Talos support matrix on minor version
The Talos<->Kubernetes support window is declared per Talos minor release, and patch releases never move it. Keying the chart's matrix dict on the full patch version meant every Talos patch bump fell out of the dict and silently disabled the render-time guard — precisely the mismatch the guard exists to catch. Derive the minor from talos.version before the lookup, so the guard keeps firing across patch releases of a known minor. Unknown minors are still skipped, leaving compatibility to the operator who overrode the pin. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
4a2e43969b |
fix(ci): polish cut-prerelease per re-review nits (#3253)
Cosmetic follow-ups from the re-review (all fail-closed, no behavior change): - Write-access gate: drop the inert 'maintain' from the allowed list. getCollaboratorPermissionLevel's `permission` field returns only the legacy roles admin/write/read/none (maintain collapses to write), so admin+write is the full "can push" set; 'maintain' never matched. - Stale-tip guard: resolve the remote tip without a pipe masking ls-remote's exit code, mirroring the write-once query — a transport failure now reports a distinct error instead of "branch not found". - tags.yaml: the hand-pushed-stable rejection message now says stable tags are finalized by the promote flow at the merge commit (pull-requests-release.yaml), not "come from promote-rc.yaml" — promote-rc opens the PR; finalize cuts the tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
31cda39dcd |
fix(ci): address review findings on cut-prerelease (#3253)
Non-blocking review notes from #3253: - Require write access to dispatch. workflow_dispatch already needs write access to trigger, but add an explicit getCollaboratorPermissionLevel gate (admin/maintain/write) as auditable defense-in-depth that fails closed. - Constrain a `main` dispatch to a new minor's pre-release (patch Z == 0). A patch-line pre-release built from main content would produce a wrong-tree release-X.Y.Z-rc.N staging branch; it must be cut from its release-X.Y line. - Guard against a stale checkout tip. workflow_dispatch pins the dispatch-time SHA, not the push-time branch tip (the gap includes the concurrency-queue wait); if the branch advanced, the push would tag a stale commit and yield an empty base_ref, failing tags.yaml after burning the write-once name. Compare local HEAD to the remote branch tip and fail with a re-run hint instead. - Distinguish ls-remote "tag absent" (exit 2) from a transport/auth failure (any other non-zero), so an infra blip cannot read as "free to create". - Shallow checkout (fetch-depth: 1, no fetch-tags) instead of a full-history clone to push one ref; drop the redundant local tag pre-check. - Run pure-input validation (tag + dispatch branch) before minting the token or checking out, so a bad dispatch fails in seconds. actionlint + zizmor clean; run blocks bash -n checked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
bece285678 |
ci: pin action refs flagged by the zizmor gate (#3223)
The zizmor "Audit workflows" gate (added in #3223) is red on main: five actions/* references in the release workflows from #3017 are pinned to a floating tag instead of a commit SHA (blanket unpinned-uses policy, High severity): - nightly.yaml (x3) actions/checkout@v4 - promote-rc.yaml actions/checkout@v4 - retention.yaml actions/create-github-app-token@v1 Pin each to the SHA the rest of the repo already standardizes on: - actions/checkout -> 34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - actions/create-github-app-token -> d72941d797fd3113feb6b93fd0dec494b13a2547 # v1 `zizmor --offline .github/workflows/` now reports no findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
47eec0157f |
feat(release): add Cut Pre-release Tag workflow (#2677)
Pre-release tags (vX.Y.Z-alpha.N / -beta.N / -rc.N) were cut by a manual `git push origin HEAD:refs/tags/<tag>`. That leaves the immutable-tag work in #2677 only partly enforceable: a repo ruleset cannot restrict `v*` tag *creation* to CI while still letting maintainers hand-push pre-releases (fnmatch has no way to match "v* except v*-rc.*"), so "do not hand-cut stable tags" stays a matter of discipline backed only by an in-workflow guard. Add cut-prerelease.yaml — a workflow_dispatch that becomes the sole entry point for creating a pre-release tag: - Generates the CozyStack CI app token (same app as promote-rc.yaml) and pushes the tag as the app, so tags.yaml fires (a tag pushed with the default GITHUB_TOKEN would not trigger it) and the push-event base_ref is populated (tags.yaml's Get-base-branch step needs it). - Validates ^v\d+\.\d+\.\d+-(alpha|beta|rc)\.\d+$ and rejects stable vX.Y.Z — stable tags are cut write-once by the promote flow at a PR merge commit. - Enforces dispatch from main or the matching release-X.Y branch, and refuses to move an existing tag (write-once). - Pushes HEAD:refs/tags/<tag> from the branch tip — the exact form docs/release.md prescribes — on ubuntu-latest (no build runner needed). With every pre-release cut by CI, repo admins can then enable a "Restrict creations" tag ruleset on `v*` with the CI app as the sole bypass actor, turning tag-creation discipline into hard enforcement. Also update tags.yaml's two human-facing error messages and docs/release.md (the RC and patch flows, the workflow list, and a new repo-side tag-ruleset enforcement note) to point at the workflow instead of a manual push. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
1f84d2d49e |
Merge origin/main into feat/e2e-chainsaw-example (post-#3244)
Fold the in-sandbox Talos image cache work into the Chainsaw layout: the new shared helper relocates to hack/e2e-chainsaw/_lib/ talos-image-cache.sh and run-kubernetes.sh sources it from there (the ouroboros DNS-check no-retry rework merges clean). The OIDC bats are resolved as deleted again; their Chainsaw fixtures deliberately skip the talos.imageFactoryURL injection — the helper only emits it when the cache is up, and the zero-worker render-side tests instantiate no DataVolume imports for it to redirect. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
scooby87
|
f91841fce6 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215)
## What this PR does Addresses #3181, where platform log ingestion could fail **silently**: fluent-bit shipped to `vlinsert-generic` while the monitoring `HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName` in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary layers make the failure loud instead of silent: 1. **Readiness gate** — `packages/extra/monitoring/templates/helmrelease.yaml` now sets a `poller` `waitStrategy` and a `healthCheckExprs` entry on the `VLCluster` `status.updateStatus`. A missing/unhealthy log store makes the `monitoring-system` release **not-ready** instead of flipping `Ready` the moment the CR is applied. Mirrors the existing seaweedfs CNPG pattern. 2. **No empty-log-store hole** — `packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render on an empty `logsStorages`. With no entry no `VLCluster` exists, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is now a loud misconfiguration. 3. **No silent fallback** — the `cozy-monitoring → tenant-root` `vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are created only when the root tenant hosts monitoring. The template reads the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value `monitoring.rootEnabled` (via `_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first render). The default preserves the historical always-create behaviour. Covered by helm-unittest across all four affected charts (141 tests): readiness-gate fields, empty-vs-populated `logsStorages` contract, `ExternalName` present/absent fallback contract, and the platform wiring into the `cozystack-values` Secret. Also adds the missing `test:` target to `packages/extra/monitoring/Makefile` so its suite runs in CI. **Known limitation / follow-up:** the legacy platform monitoring release installed directly into `cozy-monitoring` via the `cozystack.monitoring` PackageSource cannot carry the readiness gate, because the engine's `ComponentInstall`/HelmRelease builder and the vendored `helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. ### Release note ```release-note fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform setting (`monitoring.rootEnabled`) to control whether root-level monitoring is enabled. * Monitoring redirect ExternalName services are now rendered only when root monitoring is enabled (defaulting to enabled). * **Bug Fixes** * Ensures an explicitly set `monitoring.rootEnabled=false` is preserved when generating platform configuration. * Prevents incomplete monitoring deployments by failing rendering when `logsStorages` is empty. * Improves VictoriaMetrics readiness by delaying “ready” until the operator reports an operational status. * **Tests** * Added/expanded Helm-unittest suites covering monitoring wiring, redirects, readiness gating, and `logsStorages` requirements. * Added a `make test` target to run chart unittests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
a53d64d67a |
ci(e2e): move install gate to 32vCPU runner to end CPU oversubscription (#3251)
## What this PR does The E2E `Install Cozystack` gate flakes intermittently: a platform HelmRelease/Deployment sticks `InProgress` → `Failed` and its dependents cascade to `DependencyNotReady`, on a **different component each run** (observed: opensearch-operator-controller-manager, monitoring-agents-kube-state-metrics, kamaji readiness "context deadline exceeded"). That non-determinism is the signature of host resource thrash, not a product bug. **Measured envelope.** The sandbox boots three QEMU guests (`hack/e2e-prepare-cluster.bats`), each `-smp 8 -m 24576`: | Resource | Guests demand | 24-vCPU runner (current) | 32-vCPU runner (this PR) | | --- | --- | --- | --- | | vCPU | 3 × 8 = 24 | 24 → 0 host headroom | 32 → 8 (25%) headroom | | RAM | 3 × 24 = 72 GiB | 96 GiB → 24 GiB headroom | 128 GiB → 56 GiB headroom | On the 24-vCPU runner the guests alone pin all 24 host vCPUs, leaving nothing for QEMU's own emulation/IO threads, vhost-net, the host `iptables MASQUERADE` every guest image pull traverses, containerd, the kernel and the runner agent. Under the peak concurrency of the platform install the guest vCPUs contend with those host threads and lose cycles to CPU steal, so in-guest kubelet/apiserver readiness/liveness probes miss their deadlines and the HelmRelease flips. Across sampled failed runs the symptom is always a probe/readiness timeout with `DependencyNotReady`, never `OOMKilled`/`Evicted`/`MemoryPressure` — CPU steal, not memory, is the binding constraint (RAM already keeps 24 GiB of headroom). **Fix.** Move the job to the `oracle-vm-32cpu-128gb-x86-64` class (already referenced in-repo as the commented alternative). Guest sizing is unchanged (24 vCPU / 72 GiB), so the CPU-bound install does not slow down, while the host keeps 8 vCPU + 56 GiB of headroom that absorbs the emulation/network overhead the probes were starving against. This is deliberately **not** a retry or timeout bump (forbidden by `docs/agents/e2e-testing.md`): the install gate keeps its teeth, runs once, and still fails loud on a genuine product bug. It removes the harness oversubscription that manufactured false failures, and is complementary to the per-component probe-hardening work and to the in-sandbox image-mirror change (which relieves the distinct storage-import axis). **Trade-off / caveat.** The 32-vCPU class costs more per E2E run and adds load to that runner pool — an infra/cost decision for maintainers. A zero-cost alternative (drop guest `-smp` to 6 → 18 vCPU / 6 host headroom on the existing runner) was considered and rejected as primary: it cuts guest compute 25% on an already CPU-bound install and could trade steal-flakes for wall-clock-timeout flakes, unverifiable without a CI run. ### Screenshots N/A — CI-only change, no user-facing surface. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Increased the default runner size for end-to-end pull request checks. * Added clearer workflow guidance to reduce flaky readiness and liveness failures during installation steps. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0814586f0f |
feat(apps/kubernetes): overridable Talos image factory URL + in-sandbox e2e mirror (#3244)
## Problem `kubernetes-*` e2e tests fail intermittently and each take 18–22 min. Root cause: every tenant worker VM imports its Talos OS disk directly from the **public** `factory.talos.dev`. That endpoint has no byte-range support (CDI falls back to a full copy-to-scratch) and its egress stream-resets / stalls mid-transfer from the CI runner, so a worker DataVolume import hangs past the chart's 12-minute node-join deadline → 0 tenant nodes join → test fails. Seen on run `28924455578` (`kubernetes-previous`) and, on an unrelated branch, `28929095889` (`kubernetes-latest`) — so it's a shared external-dependency flake, not a per-PR bug. ## Changes **1. Overridable image source (chart).** Two new `talos.*` values, both defaulting to the public factory so rendered output is **byte-identical** and behaviour is unchanged: - `talos.imageFactoryURL` (default `https://factory.talos.dev`) — base URL for the worker OS disk image streamed in by CDI. - `talos.installerRepository` (default `factory.talos.dev/installer`) — OCI repository prefix for the in-guest `talos-reconcile` installer image. This closes the README's Phase-2 "not currently overridable" gap for air-gapped / mirrored deployments. **2. In-sandbox e2e mirror.** A best-effort `talos-image-cache` Deployment seeded once at install time with an aggressive `curl --retry` loop, then serving the image locally over HTTP with range support. Tenant Kubernetes CRs point `spec.talos.imageFactoryURL` at its Service via a shared helper that **gates on the mirror being Available and falls back to the public factory otherwise** — so the mirror can only help, never make CI worse. Deployed early so the seed overlaps the Cozystack install; readiness is resolved once and cached across bats files. ## Verification - `helm unittest` — 173/173 pass (added override + trailing-slash cases). - `helm template` — default render byte-identical to pre-PR; override redirects both the disk URL and the installer image. - Regenerated `values.schema.json`, aggregated-API Go types, `kubernetes-rd` cozyrds schema and README via `make generate` (clean, no key-reorder churn). - Manifest YAML + shell syntax validated; CR heredoc renders valid YAML in both mirror and fallback cases. ## Test plan - [ ] CI e2e: `kubernetes-latest` / `kubernetes-previous` import from the local mirror and join well within the node-join deadline. The mirror half can only be fully validated in a real CI run (busybox httpd / range serving / readiness gating); the fallback keeps it safe regardless. ```release-note feat(apps/kubernetes): the Talos worker OS image and installer are now overridable via `talos.imageFactoryURL` and `talos.installerRepository` (both default to the public factory.talos.dev), enabling air-gapped / mirrored deployments and an in-sandbox caching mirror for the e2e suite. ``` Refs #3231 — **do not auto-close**: this addresses the external-image-pull flake; the `MachineDeployment.spec.replicas` / LINSTOR-pressure concern in #3231 is a separate change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable Talos worker image and installer settings (`talos.imageFactoryURL`, `talos.installerRepository`) for Kubernetes deployments. * Added support for using a local Talos image cache during end-to-end runs when available. * **Bug Fixes** * Reduced e2e flakiness by avoiding reliance on public Talos image downloads. * Normalized Talos factory URLs to handle trailing slashes correctly. * **Documentation** * Updated Kubernetes app docs to describe the new Talos options and Phase 1 behavior changes. * **Tests** * Added/expanded e2e and template rendering tests for the new Talos image URL behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
c32ac68ab8 |
test(apps/kubernetes): anchor the trailing-slash imageFactoryURL assertion
Match the full rendered worker-disk URL shape (version + artifact + end anchor) like the sibling override assertion, so the trailing-slash normalization case cannot pass on a partial match. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
336f873b07 |
fix(e2e): serve the Talos image cache with a range-capable HTTP server
The talos-image-cache serve container ran `busybox httpd`, but the pinned alpine/k8s image ships no `httpd` applet and no standalone httpd, so the container exited 127 and crash-looped and the Deployment never became Available. The resolver then burned its full 12-minute rollout wait before falling back to the public factory — a fixed latency tax on the first kubernetes-* test while the flake the mirror targets stayed live. Serve the seeded image with a small range-capable server on the image's bundled python3 instead (stock `python3 -m http.server` ignores Range headers and answers 200, not 206). It needs no extra package pull, so the mirror comes up deterministically and keeps CDI's fast range import. The seed curl now also aborts and retries a stalled transfer (--speed-time/--speed-limit), and the resolver verifies a 206 range response before pointing tenants at the mirror. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f60fe2270b |
Merge remote-tracking branch 'origin/main' into feat/kubernetes-talos-image-factory-override
Signed-off-by: Aleksei Sviridkin <f@lex.la> # Conflicts: # packages/system/kubernetes-rd/cozyrds/kubernetes.yaml |
||
|
Aleksei Sviridkin
|
08ac063d6d |
ci(e2e): move install gate to 32vCPU runner to end CPU oversubscription
The three sandbox QEMU guests each run with `-smp 8`, demanding 24 vCPU in total. On the 24-vCPU runner that is a 1:1 CPU commit with no host headroom for QEMU's emulation/IO threads, vhost-net, the host NAT every guest image pull traverses, containerd and the kernel. Under the peak concurrency of "Install Cozystack" the guests lose cycles to CPU steal, so in-guest readiness/liveness probes miss their deadlines and a HelmRelease intermittently flips InProgress -> Failed with dependents cascading to DependencyNotReady — on a different component each run, the signature of resource thrash rather than a product bug. Move the job to the 32-vCPU / 128 GiB runner class. Guest sizing is unchanged (24 vCPU / 72 GiB), so the install does not slow down, while the host keeps 8 vCPU and 56 GiB of headroom that absorbs the emulation and network overhead the probes were starving against. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
scooby87
|
6c69c0055e |
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094)
## What this PR does
When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).
CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.
**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.
A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.
> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.
Fixes #3091.
### Release note
```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
a28ef9ba13 |
test(e2e): make the hairpin DNS check tolerate a tenant node recycle (#3241)
## What this PR does Makes the ouroboros hairpin DNS assertion in the `kubernetes-latest` e2e deterministic against a tenant worker-node recycle. This is a widespread pre-existing flake: it fails PRs that touch nothing virt-related and passes on re-run. ### Root cause (mechanism, from a failed run's own event stream) The assertion resolves `hairpin-*.example.invalid` from inside the tenant cluster using a single `--restart=Never` netshoot Pod, then waits for that Pod to reach `Succeeded`. The tenant crust-gather snapshot of a failed run shows the pod never ran the DNS loop at all: - `09:00:02` scheduler: `Successfully assigned default/dnscheck to ...-md0-vqcs7-hhfxx` - `09:00:06` kubelet on `hhfxx`: `Pulling image "nicolaka/netshoot:v0.13"` (no subsequent `Pulled`/`Started`) - node `hhfxx` then disappears from the tenant node list; on the management cluster its Machine is gone and a replacement Machine is created at `09:02:01` (`Provisioning`), while the other worker (created `08:43:39`) survives - `09:02:47` `taint-eviction-controller`: `Cancelling deletion of Pod default/dnscheck` The tenant's CAPI `MachineHealthCheck` remediates a node whose `Ready` goes `Unknown` for just 30s by deleting its Machine, KubeVirt VM and Node and provisioning a replacement. On the memory-constrained single-node CI sandbox a worker VM losing its kubelet heartbeat briefly is routine. A bare Pod bound to the deleted node is removed by the node controller and, with `restartPolicy: Never`, is never recreated — so the probe silently vanishes while its (large) image is still pulling and it has never left `Pending`. The single-shot wait then times out and reports `dnscheck pod did not reach Succeeded phase (last seen: <empty>)`, which is indistinguishable from a real DNS-resolution failure. ### The fix Wrap the probe create-and-wait in a bounded outer deadline (7m) that recreates the Pod when it is confirmed `NotFound` — the node-recycle signature — so the check is retried on a surviving node instead of failing the whole test. An empty phase is treated as ambiguous (a transient API blip is not misread as a lost Pod); only a definitive `NotFound` triggers a recreate. A Pod that actually runs to a non-`Succeeded` terminal state, or the deadline elapsing with no successful run, still fails the test. The in-Pod DNS resolution loop that the assertion exists to exercise is unchanged, so the reconciliation coverage is preserved. This fixes the mechanism at the right layer: the check tolerates a transient node recycle of a throwaway diagnostic pod rather than reporting it as a DNS regression. It is deliberately not a timeout bump — the pod was actively deleted, not merely slow, so more time alone would not help. ### Out of scope The `kubernetes-previous` failure seen in the same run is a different mechanism: both tenant worker VMs stayed `Provisioning` for 12m (DataVolume/CDI storage-import stall on the management cluster), so no tenant node ever joined and the run fails at the existing node-join diagnostic block, well before this assertion. Both smell like the same underlying single-node-sandbox resource/storage pressure, but that is an infrastructure-capacity concern, not a harness race this change can fix. Left for a separate follow-up. ### Screenshots N/A — CI/test-harness change, no UI impact. ### Release note ```release-note test(e2e): recreate the tenant hairpin DNS probe pod on a bounded deadline so a worker-node recycle mid-check is retried instead of reported as a spurious DNS failure ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability of the Kubernetes end-to-end DNS check. * The DNS validation now retries more safely during infrastructure-related pod restarts or node churn. * Failures now surface faster with clearer diagnostic output when the check does not pass. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Kirill Ilin
|
88c0cc36ad |
feat(keycloak): add Vault Kubernetes and AppRole auth for the encryption proxy (#3239)
## What this PR does Makes the optional `keycloak-kms-proxy` PII-encryption feature production-usable. **Vault auth** — new `encryption.kms.vault.auth` switch: - `kubernetes` — **recommended**. The proxy logs in with its own dedicated ServiceAccount (`keycloak-kms-proxy`; bind the Vault role to `system:serviceaccount:<ns>:keycloak-kms-proxy`), so no credential is stored in a Secret. - `approle` — `KKP_VAULT_ROLE_ID` / `KKP_VAULT_SECRET_ID` from `encryption.kms.vault.appRole.secretName`. - `token` (default, unchanged) — Vault token from `tokenSecretName`. The proxy pod runs under a dedicated ServiceAccount (never the namespace default), with its token mounted only for Kubernetes auth. **Shared DEK set** — `encryption.deksetSecretName` → `KKP_DEKSET_FILE` (mounted read-only), and `encryption.replicas` is now configurable. Without a dekset the proxy mints an ephemeral DEK (only safe on a fresh, empty database, single replica). A shared wrapped DEK set — minted by the proxy's backfill tool — is required to migrate an existing populated Keycloak and to run HA; a guard rejects `replicas > 1` without one. Render-time validation requires each mode's settings and rejects unknown values. helm-unittest covers all auth modes, the ServiceAccount/token toggle, and the dekset/replicas paths. Uses `keycloak-kms-proxy` v0.2.0. ### Release note ```release-note feat(keycloak): Vault Kubernetes/AppRole auth, a dedicated ServiceAccount, and a shared DEK set (migration + HA) for the optional PII-encryption proxy ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded Keycloak KMS proxy Vault authentication to support **token**, **AppRole**, and **Kubernetes** modes. * Added **multi-replica/HA** support via configurable DEK-set reuse, including selecting a custom DEK-set key. * Introduced a dedicated proxy ServiceAccount with **conditional token automount** for Kubernetes auth. * **Bug Fixes** * Added stricter validation for missing/unsupported Vault auth settings. * Prevented Vault credential environment variables from leaking across auth modes. * Updated rendering behavior and test expectations for proxy resources (including KEK/ServiceAccount documents). * **Chores** * Updated the Keycloak KMS proxy container image version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
ff037c04d3 |
test(monitoring): reproduce the real vmselect cache PVC name in the backfill test
/branch-review flagged that the migration comment and its bats test used the simplified vmselect cache PVC name cache-vmselect-<cr>-<ordinal>, while the real name is vmselect-cachedir-vmselect-<cr>-<ordinal>. The anchored regex matches both (it suffix-matches <component>-<cr>-<ordinal>), so behaviour was correct, but the test did not exercise the production name. Feed the realistic name in the fake kubectl so the regex is validated against what vm-operator actually provisions, and reword the migration comment to describe the suffix match generically instead of asserting a specific claim-template prefix. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Алексей Артамонов
|
09c422562a |
test(monitoring): unit-test the VM/VL PVC backfill migration behaviour
helm-unittest only inspects rendered manifests, not the operator's runtime PVC labels, so it cannot prove migration 50 actually relabels the right PVCs — the exact no-op/mis-scope class of bug this PR iterated on. Add a bats test (fake kubectl on PATH, mirroring hack/kubernetes-md0-migration.bats) that pins the runtime behaviour: it relabels the monitoring cluster's vmstorage/vmselect/ vlstorage PVCs with application.name=monitoring-system, never touches the CNPG db PVCs or a user's own VMCluster, and stays a safe version-stamping no-op when the VM/VL CRDs are absent. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
f700c9043c |
fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
2a78e7b247 |
fix(monitoring): match cleanup PVCs by release label alone, drop managed-by
Re-review on #3094 (IvanHunters): the AND selector kept managed-by=vm-operator, which on the shipped vm-operator v0.68.4 is not reliably present on the storage PVCs (it lives on the StatefulSet selector/pod labels, not the claim template), making the delete fragile. Match by apps.cozystack.io/application.name=<release>-system alone: that label is stamped on the VM/VL claim templates (67b087ab) and copied onto the PVCs by IntoSTSVolume even on v0.68.4, is already release-scoped, and is absent from the CNPG db PVCs (grafana-db/alerta-db carry only policy.cozystack.io + cnpg.io/*), so those stay untouched. Regression guard now forbids re-adding managed-by to the selector. Also apply the review's discipline items: the cleanup Job keeps only before-hook-creation (a Failed run survives for post-mortem; hook-succeeded stays on SA/Role/RoleBinding, cf. #3072) and sets backoffLimit: 0 (best-effort, no retry). Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
e01b725a48 |
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so cleanup matches
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091 storage leak stayed open. Stamp the label where it actually reaches the PVCs: on spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name= <release>-system) now matches and stays scoped to this release. Verified by rendering the CRs (label resolves to monitoring-system, matching the hook). Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei non-blocking) and a helm-unittest suite asserting the label lands on all three storage claim templates. Corrected the misleading test comment about how the label reaches the PVCs. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
69c9e69ff4 |
fix(monitoring): scope VM PVC cleanup selector to this release
Address IvanHunters review on #3094: the PVC delete selector was release-agnostic (-l managed-by=vm-operator), matching the VM storage PVCs of every VMCluster/VLCluster in the namespace. If a namespace ever hosted more than one monitoring release, tearing one down would delete the other's VictoriaMetrics/VictoriaLogs storage. Scope the selector to this release via apps.cozystack.io/application.name= <release>-system, the ownership label the monitoring chart stamps on the VM CRs' managedMetadata and vm-operator propagates onto the PVCs — mirroring the merged seaweedfs hook's release scoping. deletecollection is still not needed: kubectl delete -l lists then deletes individually. Tests: assert the release-scoped selector and add a regression guard that the delete never falls back to the bare managed-by=vm-operator selector. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
b5d1279ccd |
fix(monitoring): make cleanup deletes non-blocking and guard delete scope
Address review on #3094: - Add --wait=false to both kubectl delete calls so PVC finalizers (pvc-protection on pods still terminating) can never block the post-delete hook and stall the HelmRelease teardown. - Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop. - Tests: assert the non-blocking flag, plus negative assertions that the PVC delete stays label-scoped (no --all) and the secret delete stays name-scoped (no -l/--all). deletecollection RBAC verb intentionally not added: kubectl delete -l lists then deletes individually, so the existing delete verb suffices. BREAKING CHANGE: disabling monitoring now permanently removes its VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
73937ddff9 |
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091)
When the monitoring tenant module is disabled, the vm-operator storage PVCs (VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left orphaned. Add a post-delete cleanup hook that deletes the PVCs by the managed-by=vm-operator label and the two TLS secrets by name, plus a test target and helm-unittest suite (the package had neither). CNPG PVCs are not targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only root fs, dropped caps, resource limits); failures are logged, never blocking teardown. NOTE: disabling monitoring now permanently removes its storage PVCs and the data they hold. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
f9afc73096 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Kirill Ilin
|
158e85af85 |
chore(keycloak): bump encryption proxy image to 0.2.1
Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Myasnikov DaniilandClaude Fable 5
|
dea35f7432 |
Merge origin/main into feat/e2e-chainsaw-example (post-#3233)
Main's #3233 landed mid-reconciliation and touched the two OIDC bats again: nodeGroups (and storageClass) dropped from the render-side test CRs — the schema default {} now renders one zero-replica MachineDeployment via CAPI defaulting, so the tests provision zero worker DataVolumes (#3231). Resolve the modify/delete as deleted and mirror the fixture change (and its rationale) into the ported kubernetes-oidc-system / kubernetes-oidc-customconfig Chainsaw suites. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
0c7b598304 |
fix(apps/kubernetes): honour nodeGroup minReplicas in MachineDeployment.spec.replicas (#3233)
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file `cozy_wait_linstor_pool_free` gate + orphan `default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup — deferred to a separate follow-up PR against `hack/e2e-apps/run-kubernetes.sh`). **Do not auto-close the issue on merge** — the hardening work is a distinct concern and belongs on its own tracker entry. ## Root cause chain (recap from the issue) `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — but the chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped `kubernetes-previous`, which then saw slow LINSTOR provisioning (my cozyreport from run 28843794670 shows a ~6-minute gap between the `Kubernetes` CR being created and the CDI `ExternalProvisioning` event actually firing for the worker root disk); the tenant node registered late; Cilium's CNI init missed the 12-minute node-join deadline; and the test failed with `cni plugin not initialized`. ## Changes Two edits, applied together — the chart edit is a no-op without the test edit, and vice-versa (dropping `nodeGroups` falls back to the chart helper's default `md0`, which itself has `minReplicas: 0`, so both pieces are needed for zero workers to actually render). ### 1. `packages/apps/kubernetes/templates/cluster.yaml` ```diff spec: clusterName: {{ $.Release.Name }} - replicas: 2 ``` The field is **omitted entirely** so the cluster-autoscaler owns it alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create (documented at https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling — *"if it's a new MachineDeployment or MachineSet, use min size"*), and Server-Side-Apply field-manager ownership then keeps chart upgrades from overwriting whatever value the autoscaler patched in the meantime. Emitting the field from the chart at all — even with the semantically-correct `{{ $group.minReplicas }}` — would keep the helm-controller ↔ cluster-autoscaler field-ownership fight alive: helm-controller applies with SSA + force-conflicts (https://fluxcd.io/flux/components/helm/helmreleases/), so every routine `helm upgrade` re-asserts the chart's chosen value and drains whatever the autoscaler had patched in the meantime. ### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats` Drop the explicit `nodeGroups:` block and the paired `storageClass:` from the render-side CR bodies. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing was never part of their coverage. After the chart fix, the chart helper's default `md0` (with `minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` — the resource is still emitted for API-surface completeness but never creates a KubevirtMachine, never provisions a DataVolume, and never touches DRBD. ## Verification ### helm-unittest ``` Charts: 1 passed, 1 total Test Suites: 16 passed, 16 total Tests: 171 passed, 171 total ``` No suite asserted on `MachineDeployment.spec.replicas` directly, so nothing needed updating. ### helm template — verify the omit With `tests/values/common.yaml` (which has `minReplicas: 0`), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "0"` annotation and **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 0` on Create — no worker DataVolumes provisioned until scale-out. With a user CR that sets `minReplicas: 2` (the shape the `kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "2"` annotation and again **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 2` on Create — byte-identical topology to the pre-PR chart's fresh-install state (2 workers), but without the recurring drain-on-upgrade cycle. ### Render-side render walk `helm template` of the OIDC bats CR bodies (after this PR): 1 MachineDeployment with **no chart-managed `spec.replicas`** and `min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0` on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning triggered. Test assertions on KamajiControlPlane / KeycloakClient / KeycloakClientScope / AuthenticationConfiguration Secret are unchanged. ## Upgrade impact Existing customers stop losing workers on `helm upgrade`. Prior to this PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every Helm action, so a customer running with the autoscaler at, say, `replicas: 8` would see 6 workers drained on every routine Cozystack platform bump and then wait for the autoscaler to scale back up in response to unschedulable Pods. With `spec.replicas` no longer emitted by the chart, SSA field-manager ownership preserves the autoscaler's value across upgrades unconditionally. Fresh installs with `nodeGroups: {}` (the schema default) transition from "2 workers by default" to "0 workers until scale-out". The chart helper's default md0 carries `minReplicas: 0`, which is what CAPI's defaulting webhook seeds `spec.replicas` from. The `values.yaml` / README / helper docstring / regenerated Go type doc + openAPISchema call this out explicitly; the release-note repeats it below. See finding 5 in the paired self-review for the specific `addons.ingressNginx.enabled: true` interaction operators need to be aware of on fresh installs. ## Test plan - [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing. - [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) — MD.replicas: 0. - [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2 (byte-identical to pre-PR). - [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as before), `kubernetes-oidc-{system,customconfig}` render zero workers, `kubernetes-previous` no longer flakes on LINSTOR pool pressure originating from the earlier OIDC suite. ## Notes - Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not** in this PR — worth a separate targeted change to `hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement rather than a root-cause fix. Happy to send it as a follow-up if wanted. ## Release note ```release-note fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out: - Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action. - Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Rendered `MachineDeployment` manifests no longer set `spec.replicas`, avoiding chart upgrades overwriting autoscaler-managed sizing. * **Documentation** * Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with `minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only after an autoscaler-triggering unschedulable Pod or manual scaling; updated related storage and ingress guidance. * **Tests** * Strengthened default node group rendering assertions and adjusted OIDC e2e manifests to reduce CI provisioning churn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Fable 5
|
1f9555a3c6 |
test(e2e): resolve outstanding bot review findings
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2 provisions two, only ordinal 0 was checked) - gateway: || true each delete inside the three cleanup() traps — they run under set -eu, so one transient delete failure skipped the remaining cleanup commands - bucket, backup(.disabled): guard the trap's kill against an unset pf_pid (script can exit before the port-forward starts) Not addressed on purpose: the sort(ports[*].port) suggestions on clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15 (still the latest release) mis-evaluates, which is why the suites use the ordered (ports[*].port) projection; services keep spec order. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
dd2f89d208 |
test(sdn): port the securitygroup and serviceexposure suites to Chainsaw
Main added securitygroup.bats (#2922 membership model) and serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the Chainsaw migration branch was cut. - securitygroup: two Tests — the aggregated-API projection round-trip (backing CNP marker label, empty-value membership endpointSelector, lineage fromApp + membership fromSG peers, 1:1 rule translation, the view hiding storage internals, unmarked-CNP invisibility via an error op, delete propagation) and the controller membership-label stamp/clear lifecycle. Only the attachments annotation stays a jq script (it is a JSON string). Chainsaw auto-cleanup covers every fixture even on mid-test failure — the bats had no teardown hook and leaked its fixtures. - serviceexposure: the externalIPs backend resolve/Ready/assignedIPs path with the no-MetalLB-pool negative as an error op (ordered after resolvedBackend so it cannot false-pass), and the finalizer-removal test collapsed onto the delete op itself, which waits for full disappearance. - selection: kubernetes-application now also maps to the two OIDC render-side suites (they exercise the same app chart — a gap main's bats selection had too); the new dirs enter all_apps discovery automatically. Selector unit tests extended and green (13/13 under cozytest). - README: suites list updated. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
cac07db8fc |
test(kubernetes): port the Phase-1 OIDC render-side suites to Chainsaw
Main added kubernetes-oidc-system.bats and kubernetes-oidc-customconfig.bats after the Chainsaw migration branch was cut; without a port the 260 lines of coverage would vanish with the hack/e2e-apps/ deletion. Both are light cluster e2e (apply CR -> assert rendered objects; no Kamaji bringup, minReplicas: 0), exactly Chainsaw's declarative shape: - System mode: KamajiControlPlane carries --authentication-config + the chart-owned volume/mount; the AuthenticationConfiguration Secret carries the cozy-realm issuer (regex, script step) + per-cluster audience; per-cluster KeycloakClient/KeycloakClientScope land in the cozy realm - CustomConfig: the flag is still wired; the Secret carries the tenant-supplied issuer and NOT the cozy realm (fail-loud negation); no Keycloak objects and no pre-rendered OIDC kubeconfig Secret, expressed as error ops ordered after the positive asserts so an unreconciled HR cannot false-pass them The port also sheds two latent bats bugs: cozytest defines no skip() (the Keycloak guard would have died with exit 127 had it ever fired) and the api-resources group probe exits 0 even when the group is absent — the chart capability-gates Keycloak rendering and CI always ships the operator, so the asserts are unconditional here. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |