mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-05 11:37:31 +09:00
fix/vm-instance-disk-null
2406
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
0f3d2fa3a6 |
fix(vm-instance): emit empty disk object so KubeVirt accepts disks without bus
When a disk entry has no bus set (the schema default), the previous
template rendered `disk:` with no value — YAML parses that as
`disk: null` and KubeVirt's virtualmachines-mutator webhook denies the
request with: "spec.template.spec.domain.devices.disks.disk in body
must be of type object: null".
Restructure the disk loop so the disk/cdrom value is always a valid
object: `disk: {}` when bus is empty, `disk: {bus: <x>}` when set.
Mirrors how `cloudinitdisk` already does it in the same template.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Andrey Kolkov
|
92930ad1aa |
feat(platform): add implementation of backup-strategy controller for foundationdb
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
5f83ae72a8 |
feat(platform): implement backup strategy for mariadb (#2605)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note implement backup strategy for MariaDB ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added native MariaDB backup/restore strategy using the operator-native BackupClass/Plan/RestoreJob flow with example manifests for end-to-end workflows. * **Documentation** * Added MariaDB how‑to and examples; marked legacy restic/chart-side backup parameters as DEPRECATED and documented migration guidance. * **Tests** * Added unit tests covering MariaDB backup/restore reconciliation, templating, storage handling, idempotency, URIs, and deadlines. * **Chores** * Extended controller RBAC to allow observing MariaDB operator resources. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2605) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
f38a047aa6 |
feat(kilo): add patched image build with cross mesh granularity (#2604)
## What this PR does Adds infrastructure for building a patched kilo image from upstream `main` branch source, following the same pattern as the kamaji package. The patch introduces a new `cross` mesh granularity mode — WireGuard tunnels are created only between nodes in different locations, while intra-location traffic stays on the CNI overlay. Changes: - `images/kilo/Dockerfile` — multi-stage build: fetches upstream tarball, applies patches, builds `kg`/`kgctl` binaries with CNI plugins and iptables-wrapper - `images/kilo/patches/feat-add-cross-mesh-granularity.diff` — adds `CrossGranularity` to the kilo binary - `Makefile` — new `image:` target for building the patched image, updated `update:` target to pin upstream commit SHA ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(kilo): add patched image build with cross mesh granularity support ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added "cross" mesh granularity mode for finer topology grouping, adjusted routing/encryption behavior * Added e2e tests covering cross-mode scenarios for improved reliability * **Chores** * Build automation now tracks latest main commit and derives image digests from build metadata * Introduced multi-stage container build and runtime packaging for kilo, including required networking tooling * **Configuration** * Updated default image reference to a new repository and digest [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2604) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
92d128e8ff |
feat(keycloak): support extraEnv values and userProfile customization (#2613)
## What this PR does Two related quality-of-life additions to the Keycloak system component: 1. **`keycloak` chart now accepts `extraEnv`.** Operators can append extra `EnvVar` entries to the Keycloak StatefulSet container without forking the chart. Useful when a custom theme needs runtime configuration via `kcContext.properties`, or when an extension expects environment variables. 2. **`keycloak-configure` chart now exposes a top-level `userProfile` value** that is mirrored verbatim into `ClusterKeycloakRealm.spec.userProfileConfig`. This lets operators declare custom user-profile attributes (required fields, validators, permissions, ...) declaratively, alongside the existing `login` and `smtp` blocks. Both values default to empty, so existing deployments are unaffected. ### Release note ```release-note feat(keycloak): support extraEnv values and userProfile customization ``` |
||
|
Andrey Kolkov
|
62605a198b |
feat(platform): implement backup strategy for mariadb
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Kirill Ilin
|
3e5b370f74 |
chore(keycloak): use generic example values in chart comments
Replace deployment-specific examples in the new extraEnv and userProfile docs with generic placeholders so the chart comments do not lean on any particular downstream use case. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
2261ef7e25 |
feat(keycloak-configure): plumb userProfile into ClusterKeycloakRealm
Operators can now declare custom user-profile attributes (required fields, validators, permissions, ...) via a new top-level userProfile value. The value is mirrored verbatim into the ClusterKeycloakRealm.spec.userProfileConfig managed by the EDP keycloak-operator. The default remains empty, so realms that don't need custom user profile configuration are unaffected. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
6413052abc |
feat(keycloak): expose extraEnv values to the StatefulSet
Custom themes (Keycloakify, FTL) often need runtime values from the deployment environment via kcContext.properties. Adding an extraEnv list to the chart lets operators inject those values per cluster without forking the chart. The new field appends verbatim to the container env list and accepts the full corev1.EnvVar schema, including valueFrom for secret or configMap-backed values. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Aleksei Sviridkin
|
55a98b8815 |
docs(apps): point seven app README preset tables at canonical docs
Address review feedback from coderabbitai across: - packages/apps/kubernetes/README.md - packages/apps/mariadb/README.md - packages/apps/nats/README.md - packages/apps/qdrant/README.md - packages/apps/redis/README.md - packages/apps/tcp-balancer/README.md - packages/apps/vpn/README.md Each of these READMEs left an inline 7-row preset table listing only the legacy flat names after the parameters block above already defaulted to an instance-type value (e.g. `t1.nano`). cozyvalues-gen only regenerates the `## Parameters` block, so the hand-written section drifted. Replace the inline tables with the same short paragraph already used by postgres and kafka READMEs: name the five series, note that the seven legacy names remain accepted as deprecated aliases, and link to the canonical reference at docs/operations/resource-presets.md so the full size matrix and legacy mapping live in exactly one place. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei SviridkinandMatthieu ROBIN
|
cca54d876a |
docs(operations): resource presets reference
Document the five instance-type series (t1/c1/s1/u1/m1) introduced alongside the legacy preset deprecation, the per-size CPU and memory matrix, and the 1:1 legacy-to-instance-type mapping table that powers cozy-lib aliases, migration 39, and the operator-side deprecation warning. Single source of truth referenced from: - packages/library/cozy-lib/templates/_resourcepresets.tpl - packages/core/platform/images/migrations/migrations/39 - pkg/apis/apps/presets/legacy.go Includes an explicit callout that legacy `medium` (1 CPU / 1Gi) and instance-type `*.medium` (2 CPU) are not the same — same word, different resources. Replace inline preset tables in packages/apps/postgres/README.md and packages/apps/kafka/README.md with a short paragraph that names the five series and points at the canonical doc, since the per-app inline tables listed only the legacy seven and would drift further as the canonical matrix evolves. Co-authored-by: Matthieu ROBIN <matthieu@hidora.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei SviridkinandMatthieu ROBIN
|
55850aa3d8 |
test(apps): backward-compat helm-unittest and CI invariants
Pin the 1:1 equivalence between legacy flat preset names and their
instance-type counterparts in helm-unittest for three charts that
exercise different preset shapes:
- postgres — top-level only
- clickhouse — top + nested clickhouseKeeper preset
- opensearch — top + nested dashboards preset
Each case asserts that nano <-> t1.nano, small <-> t1.small,
medium <-> c1.small, large <-> c1.medium, and micro <-> t1.micro
render the same CPU and memory.
packages/apps/{clickhouse,opensearch}/Makefile gain a `test:` target
that calls helm unittest so hack/helm-unit-tests.sh actually runs the
new suites in CI; without it the suites would land as dead weight.
hack/check-rd-presets.sh enforces two invariants in `make unit-tests`:
- every packages/system/*-rd/cozyrds/*.yaml openAPISchema carries
the canonical 47-preset enum (40 instance-type + 7 legacy);
- no chart template carries a hardcoded legacy preset literal in
a defaultingSanitize (list "<legacy>" ...) form.
Both turn an accidental drop of a per-chart Makefile step or an
accidental legacy hardcode into a CI failure rather than silent
production-time validation rejection.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei SviridkinandMatthieu ROBIN
|
f18ce1974e |
feat(platform): migration 39 converts legacy resourcesPreset values
Add a new pre-upgrade hook migration that walks every HelmRelease
spec.values and every app CR spec under apps.cozystack.io/v1alpha1, and
rewrites legacy flat resourcesPreset values (nano/micro/small/medium/
large/xlarge/2xlarge) to their instance-type equivalents (t1/c1).
Mapping is 1:1 by CPU and memory — the migration never bumps resource
usage. The same table lives in cozy-lib (legacy aliases) and in
pkg/apis/apps/presets/legacy.go.
Properties:
- Idempotent: re-running on a converted cluster is a no-op because
non-legacy values fall through the jq map_legacy else arm.
- Best-effort: a single failed patch logs a warning and continues;
cozy-lib legacy aliases keep charts rendering with identical CPU
and memory even if the migration is incomplete. kubectl patch
stderr is captured and surfaced in the WARN line so operators see
why a patch was rejected.
- Distinguishes "no items" from "kubectl errored" (RBAC stripped,
network down, CRD missing) so a broken cluster cannot look
identical to a clean no-op run and stamp the version
unconditionally.
- Dry-run: MIGRATION_DRY_RUN=1 prints the would-be patches and
leaves the version stamp at 39.
- Final summary echoed to both stdout and stderr.
- Version stamp only advances to 40 on a fully successful real run.
Bump migrations.targetVersion 39 -> 40 so the hook executes on the
next platform upgrade.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei SviridkinandMatthieu ROBIN
|
fdffedbd4b |
feat(apps): instance-type resourcesPreset across apps and extras
Rewrite resourcesPreset values in every apps/* and extras/* chart from the legacy flat names to the new instance-type form, using the 1:1 mapping: nano -> t1.nano micro -> t1.micro small -> t1.small medium -> c1.small large -> c1.medium xlarge -> c1.large 2xlarge -> c1.xlarge Each affected values.yaml carries the canonical 47-value enum (40 instance-type names plus 7 legacy aliases) so the schema accepts both forms during the deprecation window. cozyvalues-gen regenerates the matching api/apps/v1alpha1/*/types.go kubebuilder enum and defaults, the chart values.schema.json, the README parameter table, and the ApplicationDefinition openAPISchema in packages/system/*-rd/cozyrds. Three hardcoded legacy literals in chart templates migrated: - packages/apps/harbor/templates/harbor.yaml:121 (portal) - packages/apps/opensearch/templates/opensearch.yaml:43 (dashboards) - packages/system/keycloak/templates/sts.yaml:115 (heap fallback) packages/apps/foundationdb/Makefile gains the missing hack/update-crd.sh call. Without it, every regen left foundationdb-rd cozyrds stale and new FoundationDB app CRs through cozystack-api were rejected at validation for any non-legacy resourcesPreset value. Every default-rendered chart produces identical CPU and memory to the value applied before the rename, courtesy of the legacy alias table in cozy-lib and the 1:1 default mapping. Co-authored-by: Matthieu ROBIN <matthieu@hidora.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei SviridkinandMatthieu ROBIN
|
af86c9d5b8 |
feat(cozy-lib): instance-type preset names with five series and legacy aliases
Restructure the resource preset table into a flat <series>.<size> naming convention inspired by KubeVirt and cloud-provider instance types. Five series cover the full CPU-to-memory ratio range: t1 (Tiny, 1:0.5) — burstable, low memory c1 (Compute, 1:1) — CPU-balanced, CPU-bound workloads s1 (Standard, 1:2) — proxies, caches, lightweight services u1 (Universal, 1:4) — databases and messaging m1 (Memory, 1:8) — search and analytics Each series provides eight sizes (nano, micro, small, medium, large, xlarge, 2xlarge, 4xlarge), yielding 40 presets in total. Ephemeral storage remains 2Gi for every preset. The seven legacy flat names (nano, micro, small, medium, large, xlarge, 2xlarge) are retained as backward-compatibility aliases returning their original CPU and memory values. Existing HelmReleases and app CRs continue to render identical resources without a migration step. The legacy block is deprecated and will be removed in a future release; the 1:1 conversion table is: nano -> t1.nano (250m / 128Mi) micro -> t1.micro (500m / 256Mi) small -> t1.small (1 / 512Mi) medium -> c1.small (1 / 1Gi) large -> c1.medium (2 / 2Gi) xlarge -> c1.large (4 / 4Gi) 2xlarge -> c1.xlarge (8 / 8Gi) Also fix the preset helper docstring to pass the preset string straight to unsanitizedPreset, replacing the prior list applied-to-tuple form that silently invoked the dot value as a function. Co-authored-by: Matthieu ROBIN <matthieu@hidora.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Arsolitt
|
0be25ce59e |
build(kilo): pin patched image to cozystack registry
Point values.yaml to ghcr.io/cozystack/cozystack/kilo with the freshly built image digest, replacing the upstream ghcr.io/squat/kilo reference. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
e9c3817f4f |
fix(kilo): remove stale Alpine repo override in runtime stage
Address review feedback from gemini-code-assist on Dockerfile:36: Remove custom Fastly CDN repo pinned to v3.20 while base image is alpine:3.23 — use default repos from the base image instead. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Andrey Kolkov
|
250e1b3376 |
feat(platform): implement backup strategy for clickhouse application
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Arsolitt
|
269fe97e34 |
feat(kilo): add patched image build infrastructure
Add Dockerfile for building kilo from upstream main branch source with patch support, CNI plugins, and iptables-wrapper. Update Makefile with `image:` target for building the patched image and an updated `update:` target that pins the upstream commit SHA. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
6f83d9a794 |
feat(ouroboros): bump to v0.7.2 with debug logging on tenants
The v0.7.2 ouroboros release adds --log-level / OUROBOROS_CONTROLLER_LOG_LEVEL, plants Debug logs at every informer event handler (AddFunc/UpdateFunc/ DeleteFunc with kind+namespace/name via describeObj) and at reconcile entry/exit, wires SetWatchErrorHandler on Ingress / Gateway / HTTPRoute informers, and surfaces cache-sync elapsed at Info. The diagnostic surface targets the kamaji-tenant-via-konnectivity flake where bats sees a fresh Ingress create succeed against the tenant apiserver but the controller's coredns-custom rewrite lands roughly 7-10 minutes later — without per-event visibility the failure mode is indistinguishable between 'watch never fires', 'reconcile silently errors', and 'API list returns empty until reflector reconnect'. Bumps the chart-manifest digest, the controller-image digest, and the cozystack-side image-pin assertion in lockstep. The bats deadline for the coredns-custom rewrite check rises from 180s to 900s — even on the broken-watch path the reflector eventually reconnects within that budget, so the test stops failing on the symptom while we keep the diagnostic on to learn what the symptom actually is. cozystack tenants force controller.logLevel: debug via the defaultOuroborosValues helper; operators that want quieter logs can override through addons.ouroboros.valuesOverride. Two helm-unittest fixtures pin both the default and the override path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d6c24dd078 |
fix(ouroboros): pin tenant clusterDomain to cluster.local
The cozystack tenant ouroboros wrapper deliberately left `controller.clusterDomain` empty so the upstream chart's per-pod `/etc/resolv.conf` auto-detect could pick up whatever the tenant CoreDNS actually serves. In a cozystack tenant that auto-detect resolves to the host platform's `cozy.local` rather than the tenant's `cluster.local`: the cozystack tenant kubelet injects the host's cluster-domain into pod search lists, but Kamaji's TenantControlPlane is configured with `networkProfile.clusterDomain: cluster.local` and the tenant CoreDNS Corefile serves only that domain. The mismatch makes the proxy compose its `BackendHost` as `ingress-nginx-controller.cozy-ingress-nginx.svc.cozy.local`, which NXDOMAINs against tenant CoreDNS — `/readyz` returns 503 forever, the proxy Deployment never reaches Ready, helm install times out at 10 minutes, and the e2e job hits the 2h cancel without `ouroboros.bats` ever progressing past the first kubectl wait. Pin `controller.clusterDomain: cluster.local` in `defaultOuroborosValues` so the chart bakes the resolved BackendHost against the domain tenant CoreDNS actually serves. This is the contract Kamaji TCP defines for every cozystack tenant; operators on a non-default tenant CoreDNS can still override via `addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain`. The underlying tenant kubelet injecting the wrong cluster-domain into pod resolv.conf is a separate bootstrap concern that should be addressed on its own track; pinning the chart input here keeps ouroboros operational on the default cozystack tenant topology in the meantime. Update the corresponding helm-unittest to assert the new contract. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
99e7070e4e |
fix(seaweedfs): wait for ingress-nginx-system before creating Ingress (#2406)
## What this PR does Stacks on top of #2258. Adds `dependsOn: ingress-nginx-system` to the inner `seaweedfs-system` HelmRelease so flux blocks its install until the ingress chart is Ready. ## Why In #2258's E2E run (and the `cozyreport-2258` bundle from this morning), `seaweedfs-system` failed to install with: ``` Helm install failed for release tenant-root/seaweedfs-system: 1 error occurred: * Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://root-ingress-controller-admission.tenant-root.svc:443/networking/v1/ingresses?timeout=10s": dial tcp 10.96.33.42:443: connect: operation not permitted ``` ### Root cause class The outer tenant HelmReleases (`apps/tenant/templates/*.yaml`) render one inner HR per subsystem and let flux reconcile them. Both `seaweedfs` and `ingress` outer HRs get applied in parallel, and each of them spawns its own inner HR — which flux then installs in parallel too. When `seaweedfs-system` gets to the `creating 39 resource(s)` step, it tries to create the tenant Ingress. The apiserver calls the ingress-nginx `ValidatingWebhookConfiguration` (which uses `failurePolicy: Fail` by default). If the nginx controller pod/Service/endpoint state hasn't finished reconciling on the CNI yet, the connect to the ClusterIP fails with `EPERM`, the webhook errors, Helm aborts, and flux rolls back. E2E recovers via the `flux reconcile --force` retry loop in `hack/e2e-install-cozystack.bats`, but it's flaky and adds minutes. ### Fix Introduce an explicit ordering dependency at the inner HR layer: ```yaml {{- if $ingress }} dependsOn: - name: ingress-nginx-system {{- end }} ``` Flux waits for `ingress-nginx-system` to reach `Ready=true` before installing seaweedfs — which, under Flux's wait semantics, means the ingress-nginx Deployment is Available and its webhook endpoint is routable. The race disappears. Guarded by `{{- if $ingress }}` so this only kicks in for tenants that actually have ingress enabled. ## Test plan - [ ] Rerun E2E on #2258 — `seaweedfs-system` should install on the first try without a `Retry install` iteration. - [ ] Verify rendered template still matches expectations when `$ingress` is unset (no `dependsOn` block). ### Release note ```release-note [seaweedfs] Make tenant seaweedfs wait for ingress-nginx to be Ready before creating its Ingress resource, avoiding a webhook race that intermittently failed install on fresh clusters. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved deployment sequencing so the system now waits for the Ingress controller to be ready before creating or updating resources that include Ingress. This prevents premature deployments and related failures, reduces rollout errors, and provides more reliable, predictable deployments for applications relying on Ingress resources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
45e21a330f |
feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561)
## What this PR does Adds [`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0) — a Go reimplementation of [`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy) — and wires it into cozystack as a *conditional default*: a single `publishing.proxyProtocol: true` switch turns PROXY-protocol on at the host ingress-nginx **and** auto-deploys ouroboros to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arriving at ingress-nginx without the PROXY header it now requires). Default stays `false` — clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. The chart is platform-agnostic at v0.7.0: the controller and proxy binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf` inside the pod and compose the rewrite target FQDN and the backend host at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on `cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`, federations on `k8s.example.com`, custom-domain RKE2/k3s — all work without operator override. Verified end-to-end on a `cozy.local` cluster: controller logs `cluster-domain=cozy.local` at startup, chart emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked `--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy ClusterIP. Operators that prefer a chart-baked FQDN can still set `controller.clusterDomain` explicitly — the explicit-override path is preserved for non-Service backends and for split-horizon DNS topologies. The chart runs in `coredns` mode for the host (the host CoreDNS is Talos-managed and does not carry an `import` directive, so the alternative would silently no-op there) and in `coredns-import` mode for tenant deployments (tenants run cozystack-coredns, which carries the import directive and ships an empty `kube-system/coredns-custom` ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the `ouroboros.override` data key without ever touching the chart-rendered Corefile, so it does not race the chart that re-templates `data.Corefile` on every Flux reconcile). ### Layout * `packages/system/ouroboros/` — vendored chart from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`: `controller.mode=coredns` (host scenario), `proxy.target.namespace=cozy-ingress-nginx` (override the chart-default `ingress-nginx` namespace; chart-default `serviceName=ingress-nginx-controller` matches as-is), Gateway-API watching left at upstream's default of `false`. `proxy.target.host` is left empty so the proxy composes the backend host at runtime. Image is digest-pinned at `0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c` and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is intentionally not mirrored under `ghcr.io/cozystack/*`. The chart manifest itself is also digest-pinned via `OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e` in the package Makefile, with a Chart.yaml-version vs. Makefile-version assert that aborts the update target on lockstep drift. Helm-unittest in `tests/` covers controller args, RBAC narrowing per mode, Gateway-API gating, runtime-composition flag emission in both `coredns` and `coredns-import` modes, image pin drift. * `packages/system/coredns/templates/coredns-custom.yaml` + `values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed `import` prerequisite, plus the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts` mount at `/etc/coredns/custom/`. Same package serves host and tenant CoreDNS, so this covers both layers without duplication. Helm-unittest pins the rendered Corefile via a snapshot so future `make update` does not silently drop the import directive. The empty rendered ConfigMap deliberately omits `data:` — Helm three-way merge has no template-side keys to compare against, so apiserver-side runtime writes (the `ouroboros.override` data key) survive every chart upgrade. The `notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the invariant. * `packages/core/platform/sources/ouroboros.yaml` — new `cozystack.ouroboros` PackageSource for the host install (`dependsOn: cozystack.networking, cozystack.ingress-application`). * `packages/core/platform/sources/kubernetes-application.yaml` — `kubernetes-ouroboros` component registered against `system/ouroboros` so the tenant HelmRelease can reference it. * `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` — tenant-side HelmRelease deployed via the tenant admin-kubeconfig, mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues` helper flips `mode` back to `coredns-import` for tenants and mirrors per-tenant `addons.gatewayAPI.enabled` into the controller (forcing `--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the controller on `WaitForCacheSync`). `controller.clusterDomain` is intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from `/etc/resolv.conf` and composes the rewrite target at runtime, so the same wrapper renders correctly on `cluster.local`, `cozy.local`, or any custom tenant cluster-domain. Operators on a non-default cluster-domain can still override via `valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a conditional `gateway-api-crds` entry on the same gate. If `addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled` is false, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool `true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML `yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via `toString`-then-`lower`-then-string-compare; direct bool `eq` would error on type mismatch when an operator passes the value as a string from `--set`, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. Tenant disable is single-step: flipping `addons.ouroboros.enabled: false` deletes the rendered HelmRelease, helm-controller runs `helm uninstall` on the tenant, and the chart's pre-delete cleanup hook (`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the `ouroboros.override` key on the tenant's `kube-system/coredns-custom` ConfigMap automatically. No render-time guard is wired on the tenant side — gating on a `lookup` for the leftover HelmRelease would deadlock the routine disable (the parent render runs before helm-controller applies the missing-child diff, so the lookup would always find the leftover at the moment of the flag flip). If the chart's hook fails to land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job timeout), operators recover manually via the documented tenant cleanup recipe. * `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef with `enabled` + `valuesOverride`. `make generate` re-emitted README, JSON schema, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. * `packages/core/platform/values.yaml` — new `publishing.proxyProtocol` flag with extensive comment covering the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on, otherwise external traffic breaks), the host-vs-tenant fan-out (the flag does not cascade into tenants), the Talos re-render flap window, and the asymmetric disable path. The host disable path: flipping back to `false` reverts ingress-nginx PROXY-protocol config (flip the LB off first), but the host ouroboros itself stays installed because its Package CR carries `helm.sh/resource-policy: keep` — an explicit `kubectl delete package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages` entry) actually triggers helm uninstall, at which point the chart's pre-delete cleanup hook fires and patches the host Corefile automatically. The values comment ships a manual `sed`-based cleanup recipe as a fallback for the rare case where the chart's hook fails to land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off a `lookup` against the leftover Package CR to catch operators who flip the flag back without confirming the cleanup landed. * `packages/core/platform/templates/bundles/system.yaml` — flips between the default-form and explicit-form `cozystack.ingress-application` Package emission so the host ingress-nginx gets `use-proxy-protocol` + `real-ip-header=proxy_protocol` injected only when the flag is true (`use-forwarded-headers` and `compute-full-forwarded-for` are deliberately *not* on the list — they would let any upstream proxy spoof `X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`. Host Gateway-API watching is left at the upstream chart's default of `false` because cozystack does not install Gateway-API CRDs cluster-wide on the host. * `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in tenant, controller pod running, and after creating a TLS Ingress the rewrite snippet shows up in the import ConfigMap; an in-tenant `dig` from a netshoot pod also confirms CoreDNS actually serves the rewrite — pod-readiness alone passes vacuously since the upstream chart ships no readiness probes). ### Why not KEP-1860 `ipMode=Proxy` Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB frontend entirely when `ipMode != VIP`, breaking Service ingress for L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters fronted by an external CCM-managed proxy LB. ouroboros is the correct fix for L2/BGP-announce topologies that cozystack ships by default. ### Coverage Covered by helm-unittest (host chart, coredns wrapper, tenant addon — including the `kubernetesProvider` mock that exercises the lookup-driven `proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil CI/dry-run path, the gateway-api truthy-spelling acceptance across `true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that `--proxy-service-name` + `--proxy-service-namespace` are emitted instead of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import` modes, the assertion that `--target-service-name` + `--target-service-namespace` are emitted for the proxy backend, and the `_namespace.etcd`-empty render gate) and a tenant e2e bats test that creates a real Ingress, asserts the rewrite snippet lands in the import ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry loop tolerating CoreDNS reload latency). The host-level `publishing.proxyProtocol: true` path is not e2e-tested in this PR — it requires a PROXY-protocol-aware upstream LB, which the cozystack e2e harness does not stand up. That path is exercised manually and by docs review. End-to-end runtime auto-detect was verified manually on a `cozy.local` cluster (dev17): chart deployed at v0.7.0, controller logged `cluster-domain=cozy.local`, the rewrite block landed in `kube-system/coredns` Corefile pointing at `test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig <ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first attempt, and the chart's pre-delete hook restored the Corefile cleanly on `helm uninstall`. ### Supply-chain note Image and chart are pulled from a contributor's personal namespace (`ghcr.io/lexfrei/...`). Existing precedent (`packages/system/etcd-operator/Makefile` pulls from `ghcr.io/aenix-io/...`) is similar but `aenix-io` is the cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer call. Pairs with cozystack/website#527 (operator docs). ### Release note ```release-note feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with configurable enablement and tenant overrides. * Added a packaged Ouroboros chart providing proxy, controller, CoreDNS integration, external‑DNS and etc‑hosts modes. * Introduced PROXY‑protocol gating with an explicit acknowledgement flag. * **Documentation** * Expanded addon reference and platform docs with detailed Ouroboros/PROXY‑protocol configuration and defaults. * **Tests** * Added extensive unit and e2e tests covering rendering, RBAC, uninstall cleanup, and DNS behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
8da1e7a40a |
feat(platform): implement backup strategy for postgres (#2552)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note Implemented CNPG-strategy controller for postgres backups with supporting in-place and to-copy restores. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * CNPG-based backup strategy for PostgreSQL with S3-backed object store and controller support * Option to reference existing Kubernetes Secret for S3 credentials (configurable key names) * Controller supports scheduled/ad-hoc backups and both in-place and to-copy restores; safer credential handling (no inline keys) * **Examples** * Complete manifests: bucket, credentials, CNPG strategy, BackupClass, Plan, BackupJob, RestoreJob, Postgres source/target * **Security/RBAC** * Controller RBAC expanded for CNPG resources; tests enforce no cluster-wide Secret read * **Documentation** * README, chart values/schema and UI schema updated to document secret-reference workflow and restore guidance * **Tests** * New end-to-end and unit tests covering CNPG flows, restore semantics, cleanup behavior and RBAC constraints <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
653519a472 |
fix(coredns): sanitize Chart.Version build metadata in coredns-custom label
The cozystack build pipeline tags chart versions as `0.0.0+<git-sha>`. The new coredns-custom ConfigMap interpolated `.Chart.Version` directly into the `helm.sh/chart` label, producing `cozy-coredns-0.0.0+<sha>` which the apiserver rejects: label values follow RFC 1123 and `+` is not a legal character. The chart never installed, kube-system/coredns inside tenant clusters never came up, every dependent HelmRelease stalled, and the e2e suite hit the 2h job timeout without reaching the bulk of the bats tests. Apply the standard `replace "+" "_" | trunc 63 | trimSuffix "-"` sanitization, matching the pattern the vendored upstream coredns chart already uses in its own `helm.sh/chart` helper a few directories down. Add a helm-unittest case that overrides the chart version to a build- metadata form (`0.0.0+abc1234567`) and asserts the rendered label both matches the sanitized literal and conforms to the RFC 1123 label-value regex. Without the version override the bug is invisible to the test suite — Chart.yaml ships the bare `0.0.0` placeholder and helm-unittest defaults to that, so any future template that interpolates `.Chart.Version` into a label value cannot regress past CI silently. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
026581e0e9 |
fix(seaweedfs): only depend on ingress-nginx-system when tenant owns its ingress
Address review blocker on #2406: gate the dependsOn on `eq $ingress .Release.Namespace` so sub-tenants inheriting ingress from a parent namespace don't deadlock waiting for an HR that doesn't exist locally. Also add explicit `namespace:` to match repo convention. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
43c76fd8f3 |
fix(seaweedfs): wait for ingress-nginx-system before creating Ingress
Without this, the outer tenant HelmReleases apply both seaweedfs and
ingress in parallel, which in turn creates their inner HelmReleases
(seaweedfs-system and ingress-nginx-system) in parallel too. When
seaweedfs-system tries to create its Ingress before the nginx admission
webhook's Service/endpoint state has finished reconciling on the CNI,
the apiserver's webhook call fails with:
failed calling webhook "validate.nginx.ingress.kubernetes.io":
dial tcp ...:443: connect: operation not permitted
and flux rolls back the install. E2E recovers via the "Retry install"
loop but it's flaky and adds minutes to every run.
Adding dependsOn: ingress-nginx-system on the inner seaweedfs HR makes
flux block the install until the ingress chart is Ready, which means
the controller Deployment is Available and its webhook endpoint is
routable before the first Ingress is created.
Guarded by {{- if \$ingress }} so this only kicks in for tenants that
have ingress enabled (i.e. tenant-root in the default topology).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
myasnikovdaniil
|
c22d6661ad |
fix(kafka): create LoadBalancer service when external flag is enabled (#2578)
## What this PR does When `external: true` is set in the Kafka chart, Strimzi creates a LoadBalancer service named `<release>-kafka-external-bootstrap` for external client access. However, two issues prevented this from working correctly: 1. **Unconditional listener definition**: the `external` listener was always present in the Kafka CR (as `type: internal` when disabled), instead of being omitted entirely when `external: false`. 2. **Missing dashboard RBAC**: the dashboard role only granted access to `<release>-kafka-bootstrap` (internal ClusterIP). The Strimzi-managed external bootstrap service was not included, making it invisible to tenant users even when it was created. **Changes:** - Make the external listener conditional — added only when `external: true` - Add `<release>-kafka-external-bootstrap` to dashboard RBAC rules when `external: true` - Add helm-unittest tests covering listener and RBAC behavior for both `external: true` and `external: false` ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(kafka): external listener is now correctly created as a LoadBalancer and exposed in the tenant dashboard when `external: true` is set ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * External Kafka listener and external bootstrap service in dashboard resource mapping are now conditionally enabled based on configuration settings. * **Tests** * Added comprehensive test suites validating Kafka configuration rendering and dashboard resource mapping across internal and external deployment variations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
ed5ae16e36 |
feat(platform): implement backup strategy for postgres application
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Aleksei Sviridkin
|
a9c1298b42 |
[gpu-operator] Update to v26.3.1 and add experimental vGPU variant (#2323)
## What this PR does Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an **experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and newer GPUs. Changes: - Update vendored upstream chart to v26.3.1 (CDI enabled by default, component version bumps). - Pin chart version in Makefile for reproducibility. - Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload: vm-vgpu` so the chart activates the vGPU Manager DaemonSet without per-node labelling. - Add `vgpu` variant to PackageSource alongside the existing `default` (passthrough). - Rename `values-talos.yaml` → `values-passthrough.yaml` and pin `defaultWorkload: vm-passthrough` for symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically. - Refresh `hack/dcgm-default-counters.csv` to match the new dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED` and other fields introduced upstream. `hack/check-gpu-recording-rules.bats` passes against the new fixture. - Add `hack/check-gpu-operator-variants.bats` that renders both variants via `helm template` and asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them. - Add internal documentation for the vGPU bring-up. The `vgpu` variant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed. ### `vgpu` variant is experimental NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (`echo <id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after every node reboot. Without it, `permittedHostDevices.pciHostDevices` reports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR. ### KubeVirt requirement KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which is `main`-only — released tags `v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned. ### Behavior changes inherited from upstream chart v25 → v26 The upstream chart flipped two defaults that are pinned back to `false` here, so `default` (passthrough) deployments do not silently change footprint on upgrade: - `ccManager.enabled` was flipped to `true` upstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade. - `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet would render even in passthrough mode. Both stay opt-in via Package CR overrides. The new bats test guards the pin. Other inherited changes that are user-visible and worth pinning down before upgrade: - `driver.licensingConfig.configMapName` is deprecated upstream in favour of `driver.licensingConfig.secretName`. The deprecation marker lives in the CRD's `description` field (visible via `kubectl explain`) — there is no helm-render-time warning, the chart template silently prefers `secretName` and falls back to `configMapName`. Operators with existing licensing overrides should migrate the key on the next chart upgrade. - `driver.version` upstream default jumped from `570.x` to `580.x`. Invisible on the `default` variant (which sets `driver.enabled: false`), user-visible for anyone running the upstream Ubuntu driver path. - `vfioManager.image` switched from `nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every `default` (passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters. - `dcgmExporter.image` switched to the `-distroless` flavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used to `kubectl exec` into the exporter for ad-hoc debugging will need to switch to `kubectl debug` or container ephemeral images. - `CDI` (Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is. - `node-feature-discovery` subchart source switched from the GitHub Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped installations need to mirror the new OCI URL. ### Talos Talos is **not recommended** for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (`default` variant) on Talos is unaffected. ### Release note ```release-note [gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced vGPU support variant with end-to-end configuration guide, validation procedures, and troubleshooting documentation * **Updates** * Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3 * Improved component health checks and added PodDisruptionBudget availability controls <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
8e78536cc3 |
fix(cert-manager): bump chart to v1.20.2 (#2562)
## What this PR does Bumps the vendored cert-manager chart from v1.19.3 to v1.20.2. Pure upstream bump via `make update` — no cozystack-side template changes. ### Why [Commit `2b6e20cc`](https://github.com/cozystack/cozystack/commit/2b6e20cc) migrated the platform's ACME HTTP-01 setup to the modern `ingressClassName` API on both the ClusterIssuer's solver and the per-tenant override annotation `acme.cert-manager.io/http01-ingress-ingressclassname`. The override annotation, however, was only added to upstream cert-manager in v1.20.0 ([cert-manager#8244]) — v1.19.x silently ignores it. Result: cert-manager always falls back to the ClusterIssuer's solver class (`tenant-root`), so HTTP-01 solver Ingresses for non-`tenant-root` tenants land on the wrong controller and Let's Encrypt validation fails for tenant-scoped Ingresses (grafana, harbor, alerta, etc. inside child tenants). Targeting v1.20.2 (not .0): - [cert-manager#8655] (in v1.20.1) — restores `clusterissuers/issuers/finalizers` RBAC dropped in v1.20.0. - [cert-manager#8665] (in v1.20.2) — webhook helm-template fix. ### Verified live on dev10 | Stage | `Challenge.spec.solver.http01.ingress.ingressClassName` | |---|---| | Pre-bump (v1.19.3) | `"tenant-root"` (bug) | | Post-bump (v1.20.2) | `"<tenant>"` (fixed) | Full propagation chain confirmed end-to-end: source Ingress annotation -> ingress-shim sets `http01-override-ingress-ingressclassname` on the Certificate -> acmeorders sets `spec.solver.http01.ingress.ingressClassName` on the Challenge -> http01 solver creates the solver Ingress on the tenant's class. ### Upgrade considerations - **Container UID/GID change**: 1000/0 -> 65532/65532 across controller, webhook, cainjector. PSP / SecurityContext rules pinning UID 1000 will need updating. - **Helm chart image structure** refactored from `image.repository` to `imageRegistry` + `imageNamespace` + `image.name`. Cozystack's wrapper `values.yaml` is empty, so rendered image references are unchanged (`quay.io/jetstack/cert-manager-{controller,webhook,cainjector,acmesolver}:v1.20.2`). - **NetworkPolicy templates restructured**: legacy `networkpolicy-egress.yaml` removed; per-component `networkpolicy-cainjector.yaml` and `networkpolicy-cert-manager.yaml` added. All gated on `*.networkPolicy.enabled: false` (default). No behavioral change. - **CRDs**: `selectableFields` added (Kubernetes >=1.30) and Azure DNS01 `zoneType` enum. Both pure additions. - **RBAC**: gains `clusterissuers/issuers/finalizers` and `gateway.networking.k8s.io.listenersets[/finalizers]`. - `DefaultPrivateKeyRotationPolicyAlways` feature gate moves Beta -> GA, can no longer be disabled. Already default since v1.18; no behavior change. ### Release note ```release-note fix(cert-manager): bump to v1.20.2 - fixes HTTP-01 solver IngressClass propagation for tenant-scoped Ingresses (the `acme.cert-manager.io/http01-ingress-ingressclassname` annotation was a no-op in prior cert-manager versions). Default container UID/GID changes from 1000/0 to 65532/65532 - review your PSP/SecurityContext rules. ``` [cert-manager#8244]: https://github.com/cert-manager/cert-manager/pull/8244 [cert-manager#8655]: https://github.com/cert-manager/cert-manager/pull/8655 [cert-manager#8665]: https://github.com/cert-manager/cert-manager/pull/8665 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added network policy support for controller, webhook, and CA injector * Added Azure DNS zone type selection (public/private) * Added webhook client verification configuration options * Extended pod disruption budget configuration with unhealthy pod eviction policy * **Updates** * Upgraded cert-manager to v1.20.2 * Enhanced global image registry and namespace configuration * Expanded issuer reference field selectors * Added support for extra containers in controller pod * Updated issuer terminology and configuration * **Documentation** * Updated Helm chart documentation with new configuration options * Updated post-installation guidance <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
418e3da527 |
fix(kafka): create LoadBalancer service when external flag is enabled
When `external: true` was set, the Kafka chart relied on Strimzi to create a LoadBalancer service via listener configuration, but the listener was unconditionally defined (as `type: internal` when disabled). Additionally, the dashboard RBAC did not include the Strimzi-managed external bootstrap service, making it invisible to tenant users. - Make the external listener conditional: only add it to the Kafka CR when `external: true` is set - Add `<release>-kafka-external-bootstrap` to dashboard RBAC rules when `external: true`, so the LB service is accessible from the tenant dashboard - Add helm-unittest tests for both kafka listeners and dashboard RBAC covering the external/internal paths Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
7f01c7eb32 |
feat(platform): single switch for host PROXY-protocol + ouroboros
Adds publishing.proxyProtocol on the platform values: a single flag that both turns PROXY-protocol on at the host ingress-nginx (via the ingress-application Package CR's component values dict) and emits the new cozystack.ouroboros Package CR for the host install. The two pieces are coupled by design — turning PROXY-protocol on at ingress-nginx without ouroboros breaks intra-cluster traffic to the cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits the LB IP, the connection arrives at ingress-nginx without the PROXY header it now requires, and ingress-nginx closes the connection). The new sources/ouroboros.yaml PackageSource declares the host chart with dependsOn: cozystack.networking + cozystack.ingress-application so install ordering is deterministic. The bundle template emits the host ouroboros Package via the standard helper — the parent chart's Gateway-API watching default of false stays in place, because cozystack does not install Gateway-API CRDs cluster-wide on the host (system/ gateway-api-crds is referenced only from the kubernetes-application PackageSource, tenant scope). Forcing --gateway-api on a host without the CRDs would CrashLoop the controller on WaitForCacheSync against missing GVRs. The flag default is false. Clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. Disabling: flipping publishing.proxyProtocol back to false stops emitting the cozystack.ouroboros Package CR, but every Package CR carries helm.sh/resource-policy: keep — Helm leaves the existing Package on the cluster so ouroboros stays installed and the live Corefile rewrite block keeps pointing at the still-running ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To actually uninstall ouroboros the operator removes the Package CR explicitly (or adds the package name to bundles.disabledPackages) and runs the documented Corefile cleanup recipe in the same window — otherwise stale rewrites point at a deleted Service and in-cluster DNS for hairpinned hostnames returns connection-refused. The values comment ships the cleanup recipe (a JSON merge-patch on data.Corefile that strips the BEGIN/END block while preserving labels, annotations, and other data keys including any Talos-managed metadata) and documents the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on), the host-vs-tenant fan-out (the flag does not cascade into tenant clusters; tenants opt in per-Kubernetes-CR via addons.ouroboros.enabled), and the Talos re-render flap window (machine-config pushes wipe the BEGIN/END markers; the controller re-applies on next reconcile, ≤10 minutes by default). Also drops the stale "distro-full" entry from the bundles.system.variant options comment in core/platform/values.yaml — the variant was removed in v1.0.1 and was never restored. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7ab940f309 |
feat(kubernetes): expose ouroboros as a per-tenant addon
Adds the OuroborosAddon typedef, addons.ouroboros knob, and tenant HelmRelease that deploys the cozy-ouroboros chart inside a tenant Kubernetes cluster via the host's flux-helm-controller and the tenant admin-kubeconfig — same shape as the cert-manager addon. The HelmRelease overrides controller.mode back to coredns-import because tenants run the cozystack-coredns wrapper (which carries the `import /etc/coredns/custom/*.override` directive); the parent chart's default of mode=coredns is the right shape only for the Talos-managed host CoreDNS. controller.gatewayApi.enabled is mirrored from the per-tenant addons.gatewayAPI.enabled — forcing --gateway-api on a tenant without Gateway-API CRDs CrashLoops the controller on WaitForCacheSync against missing GVRs. dependsOn grows a conditional gateway-api-crds entry on the same gate. controller.clusterDomain is intentionally NOT pre-set by the wrapper. The chart auto-detects it inside the controller pod by parsing /etc/resolv.conf — the search list of which is `*.<ns>.svc.<domain>` and reflects exactly the zone the tenant CoreDNS actually serves (governed by KamajiControlPlane.spec.networkProfile.clusterDomain defaulting to `cluster.local`, and the cozystack-coredns wrapper kubernetes-plugin zone, also `cluster.local`). Pre-setting from a host-side fact like `_cluster.cluster-domain` would override the auto-detect with a value that does not match what the tenant kube-dns serves, silently emitting NXDOMAIN-bound rewrites. Operators that run a tenant on a non-default cluster-domain can still override via addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain. If addons.ouroboros.enabled is true but ingressNginx is off, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool true, string "true"/"True"/"TRUE", numeric 1, YAML "yes"/"Yes"/"YES"/"on"/"On"/"ON"/"y"/"Y") via toString-then-lower-then-string-compare; direct bool eq would error on type mismatch when an operator passes the value as a string from --set, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. The acknowledgeUncleanRemoval gate is defence-in-depth for the rare edge case where helm uninstall fails to land cleanly (tenant unreachable, hook timeout, manual kubectl delete bypassing helm) — in the normal path the upstream chart's pre-delete cleanup hook (coredns-cleanup-hook.yaml, vendored at 0.6.0) nulls the ouroboros.override key automatically when the HelmRelease is uninstalled. The gate only blocks render when a leftover HelmRelease on the host kube-apiserver shows the previous install is still sitting on the cluster. The component is also registered in the kubernetes-application PackageSource (as kubernetes-ouroboros at path system/ouroboros) so the tenant HelmRelease's ExternalArtifact resolves. make generate re-emitted README.md, values.schema.json, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. helm-unittest covers: render gating on enabled flag, hard-fail when ingress-nginx is off, namespaces and labels, chartRef, dependsOn cilium/coredns/ingress-nginx + conditional gateway-api-crds, default coredns-import mode, valuesOverride precedence, gateway-api flag mirroring in both directions, gateway-api truthy-spelling acceptance (including case-insensitive on/On/True/TRUE), the lookup-driven acknowledge gate (kubernetesProvider mock asserts the failedTemplate path when a prior HelmRelease still exists on the host), the lookup-returns-nil CI/dry-run path (render proceeds when the cluster mock has no objects), the _namespace.etcd-empty render gate, and that controller.clusterDomain is NOT pre-set in the rendered HelmRelease (so chart auto-detect runs). Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9f8e7f9d4a |
feat(coredns): import /etc/coredns/custom into the Corefile
Extends the cozystack-coredns wrapper with the standard CoreDNS `import /etc/coredns/custom/*.override` directive in the .:53 server block and renders an empty kube-system/coredns-custom ConfigMap as a chart-managed prerequisite, mounted into the CoreDNS Deployment at /etc/coredns/custom/ via extraVolumes / extraVolumeMounts. The ConfigMap carries helm.sh/resource-policy: keep so the chart-managed empty value cannot strip what later writers add into it. This exposes a stable extension point on the cozystack CoreDNS for operators (or other cozystack components) that need to add plugin-only rewrite snippets without touching the chart-rendered Corefile — otherwise the chart would re-template data.Corefile on every Flux reconcile and overwrite the additions. The same package serves both host and tenant CoreDNS deployments, so this plumbing covers both layers without per-layer duplication. The volume mount is `optional: true` for defence-in-depth: a future chart upgrade that drops the templates/coredns-custom.yaml render still leaves CoreDNS healthy with no override snippets in play. helm-unittest fixtures pin the ConfigMap shape (name, namespace, keep annotation, no data), the `import` directive in the rendered Corefile, the presence of the `reload` plugin (required for live ConfigMap updates), and the volume + volumeMount on the CoreDNS Deployment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
28e88e9b1d |
feat(ouroboros): add cozy-ouroboros system chart
New packages/system/ouroboros/ vendors lexfrei/ouroboros 0.6.0 from
oci://ghcr.io/lexfrei/charts/ouroboros and pins cozystack-side defaults
for a hairpin-NAT fix on top of ingress-nginx + PROXY-protocol.
The controller watches Ingress (and optionally Gateway / HTTPRoute)
for public hostnames and rewrites cluster-internal DNS so those
hostnames resolve to a small in-cluster TCP proxy that prepends the
PROXY-protocol v1 header before forwarding to ingress-nginx — without
that, intra-cluster traffic to the cluster's own public hostnames
arrives at ingress-nginx without the PROXY header it requires and
ingress-nginx closes the connection.
cozystack pins:
* controller.mode=coredns by default (mutates the live
kube-system/coredns Corefile in place between BEGIN/END markers).
The host CoreDNS on cozystack is Talos-managed and does not carry
an `import /etc/coredns/custom/*.override` directive, so the
alternative coredns-import mode would silently no-op there.
Tenant deployments override this back to coredns-import via the
apps/kubernetes wrapper (separate change).
* proxy.target.host=ingress-nginx-controller.cozy-ingress-nginx.svc.cluster.local —
cozystack deploys ingress-nginx into cozy-ingress-nginx in both
the host control plane and tenant clusters; the chart-default
`ingress-nginx` namespace exists nowhere in cozystack and would
silently dial a non-existent Service.
* controller.gatewayApi.enabled defaults to upstream's false at the
parent level. The bundle template (separate change) flips it on
for the host scope; the tenant wrapper mirrors per-tenant
addons.gatewayAPI.enabled. Forcing --gateway-api on a tenant
without Gateway-API CRDs CrashLoops the controller on
WaitForCacheSync.
* image is digest-pinned at
ghcr.io/lexfrei/ouroboros:0.6.0@sha256:b33c415cd2edf231eb62076297fee1b75657bef2bfcc0be916b84088aa49479d and pulled
directly from the upstream registry; this package is intentionally
not mirrored under ghcr.io/cozystack/*. Air-gapped operators
mirror both the chart and the image separately — bump together
with OUROBOROS_CHART_VERSION in the Makefile.
The vendored chart, helm-unittest fixtures (mode dispatch, RBAC
narrowing per mode, Gateway-API gating, target FQDN), and chart
layout follow the same shape as packages/system/cilium and
packages/system/metallb.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
myasnikovdaniil
|
7520967d8d |
feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) (#2509)
## Why
The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:
- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.
This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.
## What
### New operator flags
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.
### Chart exposure
`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.
## Production safety
- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)
## Verification
- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`
## Followups (deliberately not bundled)
- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.
## Test plan
- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.
* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.
* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
086e9bb35b |
fix(installer): bootstrap cozy-system via --create-namespace + label hook (#2508)
## Why helm v3 has a chicken-and-egg with charts that ship their own Namespace resource: - **With `--create-namespace`** on the install: helm pre-creates the namespace via plain kubectl-create (no helm meta annotations); the chart's own Namespace apply then fails with `already exists`. - **Without `--create-namespace`**: helm fails immediately because it can't write its release-secret to a non-existent namespace. The cozy-installer chart shipped a `Namespace cozy-system` resource since the chart's introduction (December 2025). Every cold install since has hit this conflict. It was masked by the 3× retry on `Install Cozystack` in `.github/workflows/pull-requests.yaml`: attempt 1 fails with the conflict, attempt 2 sees the existing failed release and takes the upgrade code path which patch-merges instead of strict-create. Surfaced when retries on the install step were dropped. ## What - Remove `Namespace cozy-system` from `packages/core/installer/templates/cozystack-operator.yaml`. - Add `packages/core/installer/templates/cozy-system-labels.yaml`: a pre-install/pre-upgrade hook (ServiceAccount + ClusterRole + ClusterRoleBinding + Job) that patches `cozystack.io/system=true` and `pod-security.kubernetes.io/enforce=privileged` onto the namespace. - Hook RBAC is scoped to `verbs: [get, patch]` on `resourceNames: [cozy-system]` only. - `hook-delete-policy: before-hook-creation,hook-succeeded` so the hook surface only exists during install/upgrade. Install commands now use `helm upgrade --install --namespace cozy-system --create-namespace …` — standard pattern, matches kube-prometheus-stack / argo-cd / cert-manager. ## Verification End-to-end on a kind cluster: - Cold install: 3.3s clean, no retries needed - Upgrade (re-run): 3.0s, idempotent, namespace stays labeled - Helm uninstall + re-install: clean ## Test plan - [ ] CI on this PR passes E2E - [ ] Existing clusters: namespace already labeled correctly, hook re-asserts on next upgrade — no migration needed ```release-note fix(installer): cold installs no longer fail on the chart's Namespace resource colliding with `helm --create-namespace`; cozy-system is now bootstrapped via a pre-install hook. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an automated pre-install/pre-upgrade step that applies system identity and pod-security labels to the system namespace during deployment, run in a hardened one-shot task before normal workloads start. * **Chores** * Separated namespace initialization from the operator install flow so the operator no longer renders or retains the system namespace directly, improving installation sequencing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
90b3719f6a |
fix(mongodb): expose replica-set members per pod for external access (#2514) (#2522)
## What this PR does Fixes #2514. When `external: true` is set on a non-sharded `MongoDB`, Cozystack rendered a single `LoadBalancer` Service whose selector matched **all** replica-set members (`app.kubernetes.io/replset: rs0`). Round-robin balancing sent ~⅔ of writes to secondaries and they were rejected with `MongoServerError: not primary`. Drivers couldn't recover on their own either: connecting through the LB returned the in-cluster DNS names of all members from `isMaster`, none of which resolve from outside the cluster, so automatic primary discovery failed. This PR replaces the manual `external-svc.yaml` (for replica-set mode) with the Percona MongoDB operator's native `replsets[].expose` feature: ```yaml replsets: - name: rs0 expose: enabled: true type: LoadBalancer externalTrafficPolicy: Local ``` The operator then: - creates one `LoadBalancer` Service **per pod** (`<release>-rs0-0`, `-1`, `-2`) with `statefulset.kubernetes.io/pod-name` selectors, - watches `Service.status.loadBalancer.ingress` for each pod and rewrites the member `host` in `rs.conf` to the external IPs. External clients use a standard replica-set URI (`mongodb://...:27017,...:27017,...:27017/?replicaSet=rs0`) and the driver discovers the primary correctly from outside the cluster. For **sharded** mode the existing single `LoadBalancer` is preserved — it front-ends stateless `mongos`, which can route writes to the right shard primary itself. `external-svc.yaml` is now scoped to that case only. ### Tested - `helm unittest .` in `packages/apps/mongodb/`: 91/91 pass. - Applied on a dev stand: - `replsets[0].expose` is set as expected in the rendered PSMDB CR. - The Percona operator (`v1.21.1`) accepts the spec and creates 3 per-pod `LoadBalancer` Services with `statefulset.kubernetes.io/pod-name` selectors. The previous single round-robin Service is no longer rendered. - With an LB IP pool present, the operator promotes the allocated LB IPs into the PSMDB `status.host` field and into `rs.conf`. End-to-end mongosh write loop wasn't run on the dev stand because the underlying network does not advertise/route allocated LoadBalancer IPs (no working L2/BGP path), so the rewritten `rs.conf` members are unreachable in that environment. The structural change is otherwise fully verified and the issue's reproduction scenario expects a real LB-capable network. ### Release note ```release-note fix(mongodb): expose each replica-set member through its own LoadBalancer when `external: true`, so external clients can discover the primary via the standard replica-set connection string and writes no longer fail with "not primary" errors. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * External LoadBalancer services now only render when both external and sharding are enabled. * Replica set external exposure is now conditional based on the external setting. * Service configuration now uses LoadBalancer type with Local traffic policy for external access. * Service selectors refined for proper mongos targeting in sharded deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
0b70197bf1 |
fix(kafka): bump default resourcesPreset to medium (#2537)
## What this PR does Changes the default `resourcesPreset` for both `kafka` and `zookeeper` in the Kafka managed application from `small` (1 CPU / 512Mi) to `medium` (1 CPU / 1Gi). The `small` preset is no longer sufficient for the current Strimzi 0.45 / Kafka 3.9 image baseline: - Strimzi sets JVM heap to 50% of the container memory limit for Kafka and 75% for ZooKeeper. With a 512Mi limit that leaves only 256Mi (Kafka) or 128Mi (ZooKeeper) for the JVM non-heap. - Actual non-heap usage in the Kafka 3.9 image (Metaspace, code cache, direct buffers, JMX javaagent, embedded Jetty admin server for `/v1/ready` and KRaft migration endpoints, thread stacks) systematically exceeds those budgets at startup, triggering a cgroup OOM kill (exit 137) before the broker can connect to ZooKeeper. Result on a fresh deployment with the previous default: every Kafka broker and every ZooKeeper pod ends up in `CrashLoopBackOff`, with `lastState.terminated.reason: OOMKilled`. ZooKeeper occasionally hits `OOMKilled` once quorum traffic ramps up, breaking the quorum and causing Kafka brokers to fail with `ZooKeeperClientTimeoutException`. The `medium` preset (1Gi) is the smallest preset that gives both Kafka (heap 512Mi + ~500Mi non-heap budget) and ZooKeeper (heap 768Mi + 256Mi non-heap budget) enough headroom to start cleanly. Users who need more can still pick `large`, `xlarge`, or `2xlarge`, or override `resources` explicitly. The `small` preset is intentionally kept in the enum so existing deployments that pin `resourcesPreset: small` are not broken; only the default changes. ### Release note ```release-note fix(kafka): default `resourcesPreset` for both Kafka and ZooKeeper is now `medium` (1 CPU / 1Gi). The previous default (`small`, 512Mi) is no longer sufficient under the current Strimzi 0.45 / Kafka 3.9 image and caused OOMKilled CrashLoopBackOff on fresh deployments. Existing deployments that explicitly set `resourcesPreset` are unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated default resource allocation preset from "small" to "medium" for Kafka and ZooKeeper components across CRD type definitions, Helm chart values, and JSON schema configurations. * **Documentation** * Updated parameter reference tables to reflect the new default resource preset values for both Kafka and ZooKeeper. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
0a3a5039d4 |
fix(kubevirt-operator): move PrometheusRule to Helm-rendered template (#2543)
## What this PR does
Move `alerts/PrometheusRule.yaml` from raw-file loading (via
`.Files.Glob` in
`templates/alerts.yaml`) to a proper Helm-rendered template
`templates/prometheusrule.yaml`.
The raw-file approach silently passed unescaped `{{ $labels.xxx }}` to
Kubernetes unchanged, which worked by accident. As a Helm template,
those
expressions must be escaped with `{{ "{{" }}` to prevent Helm from
trying
to evaluate them as Go template variables.
Also removes `templates/alerts.yaml` which became dead code once the
`alerts/` directory is empty.
### Release note
\`\`\`release-note
fix(kubevirt-operator): move PrometheusRule alerts to Helm-rendered
template and escape Prometheus template variables
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Reorganized alert configuration structure for VM and VMI monitoring.
Alerts that trigger when virtual machines or machine instances remain in
a non-running state for 10 minutes are now managed through an updated
template-based approach, consolidating alert definitions while
maintaining the same monitoring behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
87f2f64414 |
chore(kubevirt-operator): update to v1.8.2 (#2502)
## What this PR does Updates KubeVirt operator from v1.6.3 to v1.8.2. This update brings: - New CRD fields for changed block tracking (required for storage-agnostic backups) - Architecture support updates (s390x support, ppc64le deprecated) - Updated operator manifest with latest capabilities The standalone ServiceMonitor and alerts templates were removed as they are now integrated into the main operator manifest. ### Screenshots N/A - infrastructure component update ### Release note \`\`\`release-note chore(kubevirt-operator): update to v1.8.2 Updated KubeVirt operator from v1.6.3 to v1.8.2, adding support for changed block tracking and latest architectural improvements. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new configuration fields for changed block tracking, feature gate management, hypervisors, and VM backups. * **Chores** * Upgraded KubeVirt operator to v1.8.2. * Updated RBAC permissions and cluster access policies. * Removed Prometheus monitoring integration and alert template processing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
730037c85d |
fix(installer): address review feedback on cozy-system bootstrap
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks. - Skip hostNetwork=true and not-ready/CNI tolerations on the hosted variant: managed clusters provide CNI before install, and kube-system may carry strict PSA there that would reject hostNetwork pods. - Add packages/core/installer/templates/cozy-system-namespace.yaml gated on .Values.bareNamespace (default false). The top-level Makefile's manifests target now passes --set bareNamespace=true and renders Namespace + operator into _out/assets/cozystack-operator-*.yaml so the kubectl-apply install path stays self-contained. Helm install/upgrade callers keep the default and bootstrap via --create-namespace + the pre-install labeler hook. - Add a bats assertion in hack/e2e-install-cozystack.bats checking cozy-system carries pod-security.kubernetes.io/enforce=privileged and cozystack.io/system=true after helm install. - Inline comment near hostNetwork=true documenting why dnsPolicy left at Default (cluster DNS not up at install time; ClusterFirstWithHostNet would not help). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
6ad5bf0bbf |
fix(objectstorage-controller): retry on Bucket conflict during BucketAccess reconcile (#2529)
## What this PR does
Carries an upstream-bound patch against COSI v0.2.2's BucketAccess
sidecar reconciler so it survives the optimistic-concurrency race with
the central Bucket reconciler running in the same process.
In `sidecar/pkg/bucketaccess/bucketaccess_controller.go`, the `Add` path
does:
```go
bucket, _ := buckets().Get(...)
// ...
if controllerutil.AddFinalizer(bucket, BABucketFinalizer) {
buckets().Update(ctx, bucket, ...) // <-- 409 here
}
```
Between `Get` and `Update` the Bucket reconciler in the same binary can
mutate the object and bump `resourceVersion`, producing:
```
Operation cannot be fulfilled on buckets.objectstorage.k8s.io "bucket-...":
the object has been modified; please apply your changes to the latest
version and try again
```
surfaced to users as a `FailedGrantAccess` event on the BucketAccess.
This patch wraps the Bucket finalizer add in
`retry.RetryOnConflict(retry.DefaultRetry, ...)` and re-`Get`s the
Bucket inside the closure so each retry mutates the freshest version.
Patch is held as `91-bucketaccess-conflict-retry.diff` and consumed at
image build time (re-introduces `git` to the source stage and the `COPY
patches /patches` + `git apply /patches/*.diff` steps that existed
before `c29d501b` dropped them when 89/90 were upstreamed). The
convention is identical: drop the local patch the moment the upstream PR
merges and ships in a tagged release.
### Release note
~~~release-note
[objectstorage-controller] Carry a patch that retries on Bucket update
conflict during BucketAccess reconcile, eliminating the "object has been
modified" FailedGrantAccess event seen in CI.
~~~
## Test plan
- [x] `make image` for `system/objectstorage-controller` and verify
`values.yaml` picks up the new digest.
- [x] Re-run the cozystack-e2e suite that emitted the
`FailedGrantAccess: ... object has been modified` event on
`tenant-test/BucketAccess/bucket-test-admin` and confirm it no longer
fires.
- [ ] Verify `kubectl describe bucketaccess` reports
`AccessGranted=true` on first reconcile, no requeue from the conflict
path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced bucket access update reliability with improved conflict
handling during concurrent operations. The system now automatically
retries operations and refetches data to prevent update conflicts.
* **Chores**
* Improved Docker build process with automatic patch application support
for source customization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
8fbbef3f02 |
fix(harbor): drive bucket-secret.yaml from values, gate HR on BucketInfo (#2528)
## Summary - `cozy-harbor/templates/bucket-secret.yaml` no longer relies on `lookup` against the BucketAccess credentials Secret. It is now driven by `.Values.bucket.bucketInfo` (a JSON string) with `dig`-based safe accessors, so a missing/empty/partial value renders nothing rather than crashing with `index of untyped nil` (the failure mode that broke harbor in CI #25049445125). - The downstream `<release>-system` HelmRelease now sources `BucketInfo` via `valuesFrom` (`valuesKey: BucketInfo`, `targetPath: bucket.bucketInfo`). With the default `optional: false`, helm-controller refuses to compose values until the COSI BucketAccess controller has populated the Secret — both gating initial reconciliation and forcing a config-digest change (and thus a helm upgrade) when the credentials arrive. Flux HR `dependsOn` cannot reference COSI resources directly, so `valuesFrom` is the correct primitive here. - `bucket.secretName` was the only consumer of the old `lookup` and is removed from both the system chart's `values.yaml` default and the apps chart's `values:` block. ## Why this shape helm-controller's upgrade trigger is digest-based over composed values + chart artifact; a `lookup` returning new data on a later reconcile is not enough to force an upgrade on its own. Routing `BucketInfo` through `valuesFrom` puts the bucket data on the digest path, so the chart re-applies the moment credentials become available. ## Test plan - [x] `helm template` of `packages/system/harbor` with `bucket.bucketInfo` unset, empty string, empty JSON object `{}`, and fully populated — only the populated case renders the `*-registry-s3` Secret; the others render nothing without erroring. - [x] `helm template` of `packages/apps/harbor` produces the expected `<release>-system` HR with the new `valuesFrom` entry and the cleaned-up `values:` block. - [ ] Re-run the harbor E2E on a fresh stand and confirm 15/15. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Harbor bucket config now comes from a centralized Secret merged into chart values; in-values bucket entry removed. * **Bug Fixes** * Deployment synchronization and Helm upgrade behavior improved by depending on the Secret’s readiness/contents. * **Tests** * Added unit tests covering when the bucket secret is rendered (or not) and a make target to run those tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7eaafe0b40 |
fix(mongodb): reclaim PVCs and operator secrets on uninstall (#2525)
## What this PR does Adds the upstream Percona finalizer pair to the `PerconaServerMongoDB` custom resource so the operator performs an ordered shutdown and reclaims persistent storage when a MongoDB release is uninstalled. Previously, deleting a MongoDB application left PVCs and operator-managed secrets orphaned in the tenant namespace, requiring manual cleanup before the same release name could be reused. The finalizers added on the CR: - `percona.com/delete-psmdb-pods-in-order` — ensures the primary is shut down last for graceful cluster termination. - `percona.com/delete-psmdb-pvc` — instructs the operator to delete PVCs and operator-managed secrets when the CR is removed. Coverage applies to all replica sets in both replicaset and sharded modes, including the config server in sharded deployments. ### Behavior change Users who previously relied on orphaned PVCs as an implicit "soft delete" for data recovery should be aware that uninstalling a MongoDB release now removes the underlying storage and operator-managed credentials. The README's new "Data lifecycle" section enumerates exactly which resources are reclaimed by the operator versus by the standard `helm uninstall`. Take a backup before deletion if data retention is required. This resolves cozystack/cozystack#2513. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note fix(mongodb): reclaim PVCs and operator-managed secrets on uninstall via Percona finalizers. WARNING: upgrading scopes the admin credentials secret per release instead of the namespace-shared default. This triggers an in-place rotation of the five operator-managed admin passwords on first reconcile. Pods are not restarted; application user passwords are unaffected. Workloads consuming `<release>-credentials` must be restarted to pick up the new admin passwords. ``` ### Out of scope / follow-up - Bats integration test would require a live cluster with the Percona MongoDB operator and belongs in the `hack/e2e-*.bats` family — left as a follow-up. - Issue #2513 mentions considering similar treatment for PostgreSQL/MariaDB/Redis/ClickHouse. Each operator has a different cleanup mechanism; that work is intentionally out of scope here and should be tracked per-app. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded uninstall and upgrade guidance: which secrets/pvc remain, cert-manager TLS secrets must be deleted manually, remediation for orphaned legacy secret, and recovery steps for stuck deletions when the operator is removed before CRs. * **New Features** * Added deletion finalizers to enforce controlled teardown. * User credentials secret now uses a deterministic, release-scoped name. * **Tests** * Added chart tests for finalizer behavior (sharded/non‑sharded) and release-scoped users secret. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
6ca5566391 |
fix(operator): handle MaxHistory=0 and cover HelmRelease spec build
Chart template now uses `ne (toString …) ""` for the five helmRelease* knobs, so integer 0 (notably MaxHistory=0 = unlimited history per Helm semantics, documented in values.yaml) is no longer silently dropped. Extract buildHelmReleaseSpec from Reconcile and add unit tests asserting Interval, MaxHistory, Install/Upgrade timeouts, Strategy=RetryOnFailure, RetryInterval, and ChartRef propagate from PackageReconciler config to the generated HelmRelease. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
85ced0d729 |
fix(installer): harden cozy-system-labeler hook and add hosted variant
Address review feedback on PR #2508: - Pin alpine/k8s by digest (matches operator image pinning style). - Add resources requests/limits and a minimal securityContext to the labeler Job: capabilities drop ALL, allowPrivilegeEscalation: false, seccompProfile RuntimeDefault. - Include hook-failed in helm.sh/hook-delete-policy across all four hook resources so a failed install doesn't leave SA / ClusterRole / ClusterRoleBinding / Job objects behind. - Add explicit hosted variant case in the env block, mirroring the pattern in cozystack-operator.yaml. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
b6badeeb96 |
fix(cert-manager): bump chart to v1.20.2
cert-manager v1.20.0 introduced the
acme.cert-manager.io/http01-ingress-ingressclassname Ingress annotation
(cert-manager#8244). Cozystack templates already emit it (since
|
||
|
IvanHunters
|
000157f9d7 |
fix(kubevirt-operator): use instant vector with for: 10m instead of max_over_time
Replace max_over_time([10m]) == 1 with an instant vector selector and a top-level for: 10m clause. The previous expression fired as long as the metric was non-running at any point in the last 10 minutes; the new expression fires only when the condition is sustained continuously for the full 10-minute window, matching the alert name semantics. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
