mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 22:13:18 +09:00
91aae36da313474e4e795104f40a9cf995271c6a
1457
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
0c17d1870c |
feat(platform): add deletion-protection guardrail via ValidatingAdmissionPolicy
Block accidental DELETE on critical platform objects (tenant-root, cozy-system, LinstorCluster, CRDs, ClusterIssuers, cozystack-version ConfigMap, cozystack-packages OCIRepository, tenant-root HelmRelease) via a label-scoped ValidatingAdmissionPolicy. No webhook DaemonSet, Service, TLS, or image — the API server evaluates the CEL expression in-process. Requires Kubernetes 1.30+. Bypass for legitimate deletes is two steps: kubectl label <kind> <name> platform.cozystack.io/no-delete- kubectl delete <kind> <name> CRDs installed dynamically by the operator get the label stamped in internal/crdinstall/install.go before the apply call. Release note: Added deletion-protection guardrail (ValidatingAdmissionPolicy) blocking DELETE on cozystack platform objects labeled platform.cozystack.io/no-delete=true. Bypass by removing the label first. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
3756f5c459 |
feat(networking): Gateway API support via Cilium (supersedes #2213) (#2470)
## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since
|
||
|
myasnikovdaniil
|
2cb16d5ae6 |
refactor(build): standardize image tagging to fix concurrent PR push conflicts (#2711)
## What this PR does Concurrent PR builds were racing in the OCIR registry. Every package Makefile tagged its images via `settag`, which on any non-release commit collapsed to `:latest` — so every image across every PR was pushed to the same tag, and parallel builds got 409s on the manifest PUT. Two outliers (`metallb`, `redis-operator`) used hardcoded version tags and collided across PRs unconditionally. This PR replaces `settag` and the `TAG=latest` fallback in `hack/common-envs.mk` with a single `image-tags` macro driven by three env vars: | Env | Meaning | |---|---| | `IMAGE_TAG` | Build-unique tag, always pushed | | `PUBLISH_VERSIONED` | Also push `:<component-version>` (when provided) | | `PUBLISH_FLOATING` | Also push `:latest` | Wired into the two workflows: - **`pull-requests.yaml`** — `IMAGE_TAG=pr-<N>-<sha>`, both `PUBLISH_*` off. Every PR build pushes a unique tag set; no collisions, no `:latest` move. - **`tags.yaml`** — `IMAGE_TAG=<ref_name>`, both `PUBLISH_*` on. Release output preserved (component versions + `:latest`) plus a uniform `:<git-tag>` handle on every image for traceability. - **Local** — defaults to `IMAGE_TAG=dev`, no flags, `PUSH=0`. All 30 package Makefiles converted to `$(call image-tags,<repo>,<versioned-tag>)`. Image consumers reference digests pinned into `values.yaml` by each Makefile, so the tag is decorative for cluster-side pulls — the build-unique tag is sufficient for E2E. Also drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler). Nothing in the repo or downstream values.yaml consumed it; the same information is preserved via the new `:<git-tag>` plus the existing `:<component-version>` tags on release. `ubuntu-container-disk` builds a separate image per Kubernetes version in a loop, so its primary tag became `:<k8s-ver>-<IMAGE_TAG>` to prevent loop iterations within a single build from pushing to the same tag. Verified by dry-running `make -n image` across representative packages (simple, dual-tag, skopeo, flux-push, loop) under all four contexts (PR / release / main / dev) — tag sets are unique and behave as expected. ### Release note ```release-note refactor(build): PR CI now pushes images under a unique `pr-<N>-<sha>` tag per build instead of `:latest`. Concurrent PR builds no longer race in the registry. Release builds keep the existing `:<component-version>` and `:latest` tags and additionally publish a uniform `:<git-tag>` handle across every image. The redundant `:<component-version>-<git-tag>` tag (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) is no longer published — image references in values.yaml are digest-pinned and were the only legitimate consumers. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Centralized and simplified container image tagging across build workflows. * PR builds now use unique image tags derived from PR metadata for better isolation. * Added configurable flags to control publishing of versioned and floating tags, making release tagging behavior more predictable. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2711?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
afc7765e69 |
ci(external-dns): regenerate cozyrds schema for gatewayAPI description
`cozyvalues-gen` bakes the values.schema.json's
`gatewayAPI.description` into the JSON openAPISchema field of the
external-dns release-definition CRD at
`packages/system/external-dns-rd/cozyrds/external-dns.yaml`. The Q7
commit (
|
||
|
Aleksei Sviridkin
|
32241afe03 |
feat(gateway): drop tenant-* ban on gateway-attached-namespaces
Removes two gates that prevented child tenants from attaching to a parent's publishing Gateway: 1. Render-time fail() in templates/gateway-hostname-policy.yaml that refused to render the chart when _cluster.gateway-attached- namespaces contained any tenant-* entry. 2. cozystack-gateway-attached-namespaces-policy VAP that enforced the same rule at Package CR admission time. The original rationale — 'a tenant in attachedNamespaces can hijack the publishing tenant's hostnames' — assumed a static-name allowedRoutes whitelist where adding tenant-* literally widened the attach surface for arbitrary hostnames. Under the inheritance refactor (Stage 1/3/5) the attach surface is governed by the namespace.cozystack.io/gateway label selector, not by entries in AttachedNamespaces; and hostname hijack is independently blocked by three other layers: * Layer 4 — cozystack-tenant-host-policy VAP on Tenant.spec.host * Layer 5 — cozystack-namespace-host-label-policy VAP on the namespace label * Layer 7 — cozystack-route-hostname-policy VAP on HTTPRoute and TLSRoute hostnames Those three defend against the hijack vector regardless of what sits in attachedNamespaces, and they apply uniformly to every tenant (inheriting or owning a Gateway). The dropped gates added no extra defence, while their presence broke the inheritance flow kvaps reasonably asks for in the May 25 review. Doc count for gateway-hostname-policy.yaml drops from 8 to 6 (one VAP + one Binding removed); helm-unittest fixtures updated to match, with a regression-guard test asserting the removed VAP name no longer appears in any rendered document. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
30302eabe4 |
feat(tenantgateway): switch Gateway allowedRoutes to label selector
The Gateway's HTTPS / TLS-passthrough listeners now match attached namespaces by the namespace.cozystack.io/gateway label keyed on the TenantGateway's own namespace, instead of a static kubernetes.io/metadata.name In [list] whitelist. This restores inheritance parity with the legacy ingress flow: child tenants of a Gateway-owning ancestor publish through the ancestor's Gateway by default — without their own LB IP / Certificate, and without operator intervention to add each child by name to platform values. Two writers populate the label: * apps/tenant chart namespace.yaml — each tenant namespace gets a label pointing at the nearest ancestor that owns a Gateway (self if owning, inherited otherwise). The tenant Helm chart is the source of truth for tenant-tree labels; the controller never touches them (annotation-gated GC). * cozystack-controller (ensureNamespaceLabels) — patches the label onto every namespace in tgw.Spec.AttachedNamespaces, so cozy-* system namespaces (cert-manager, monitoring, harbor, …) reach the publishing Gateway alongside the tenant tree. Tracks its own writes via the cozystack.io/gateway-attached-by annotation; on a reconcile where an entry is removed from AttachedNamespaces, the label is garbage-collected (annotation present + namespace no longer expected → strip both label and annotation). Also adds the gateway-attach label to tenant-root itself via cozystack-basics/templates/tenant-root.yaml when _cluster.gateway-enabled is true, so the Gateway in tenant-root's own namespace passes its own selector for the controller-owned http→https redirect HTTPRoute. Tests (TDD, red→green): * TestReconcile_HTTPSListenerUsesGatewayLabelSelector pins the new MatchLabels shape on HTTPS listeners. * TestReconcile_LabelsAttachedNamespaces pins the patch path: every namespace in spec.AttachedNamespaces (plus tgw.Namespace) carries namespace.cozystack.io/gateway=<owner> after reconcile. * TestReconcile_LabelGCRemovesDroppedAttachedNamespaces pins the GC path: removing an entry between reconciles strips the label. * TestReconcile_LabelGCDoesNotStripHelmOwnedLabels pins the safety contract: labels lacking the controller annotation (Helm-owned tenant labels) are left alone. * tenant-root_test.yaml adds the positive case for the conditional label render under _cluster.gateway-enabled=true. The HTTP listener (port 80) keeps the narrower static-name selector — it allows only the tenant namespace plus cozy-cert-manager (for HTTP-01 ACME challenges) and must NOT accept routes from app namespaces or child tenants, since unattended routes on the HTTP listener serve plaintext and leak credentials. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
0994821bda |
feat(monitoring): toggle ingress↔HTTPRoute on _cluster.gateway-enabled
Adds Gateway API equivalents for the two ingress endpoints in the
monitoring chart — Grafana (web UI) and Alerta (alert console) — gated
on the same _cluster.gateway-enabled flag the keycloak / dashboard /
cozystack-api templates already use.
* Grafana's ingress is configured inline via Grafana CRD's spec.ingress
block; that block is now wrapped with {{ if ne gateway-enabled true }}
and a standalone HTTPRoute attaching to the tenant Gateway is rendered
in the alternate branch.
* Alerta uses a standalone Ingress object; the same Ingress↔HTTPRoute
swap is wired through an if/else.
Both routes attach to Gateway cozystack in the publishing namespace
(default tenant-root), matching the convention established by the rest
of the system charts.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
039924f59d |
feat(gateway): align _namespace.gateway propagation with tenant.gatewayEffective
Two related fixes for the gap where the tenant chart's gateway HelmRelease materialises but child apps (harbor, bucket) silently fall back to ingress-nginx: - packages/apps/tenant/templates/namespace.yaml: switch the _namespace.gateway resolution from raw .Values.gateway to tenant.gatewayEffective. This is the same helper consulted by gateway.yaml (HelmRelease render), so per-tenant Gateway materialisation and child-app HTTPRoute / Ingress selection now stay in lockstep — including the auto-default for derived-apex tenants. - packages/system/cozystack-basics/templates/cozystack-values-secret.yaml: add a conditional gateway field to the tenant-root cozystack-values Secret. cozystack-basics is the only writer of _namespace.* for tenant-root (apps/tenant chart skips tenant-root via the existing ne 'tenant-root' gate). Without this line, tenant-root apps read _namespace.gateway as empty and route to ingress even when the platform's gateway.enabled flag is on. Conditional on _cluster.gateway-enabled='true' so non-Gateway clusters keep the empty value. Test fixture in cozystack-basics/tests/cozystack-values-secret_test.yaml pins the conditional and the regression-guard that the other four _namespace keys (etcd, ingress, monitoring, seaweedfs) stay unconditional. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d2e851f63b |
feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov DaniilandClaude Opus 4.7
|
1bcaa3de54 |
fix(flux-plunger): correct common-envs.mk include path
The include pointed at ../../../scripts/common-envs.mk, but that directory does not exist — only hack/common-envs.mk does. flux-plunger is not part of the root `make build`, so the broken path went unnoticed until the new image-tags macro made the recipe depend on it. Without the include, `make -C packages/system/flux-plunger image` fails immediately with "No such file or directory". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
debb9f85db |
feat(platform): add implementation of backup-strategy controller for etcd (#2641)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note add implementation of backup-strategy controller for etcd ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Cluster-scoped Etcd backup strategy CRD (S3-only) plus controller support for BackupJob (snapshot → Backup) and destructive in-place RestoreJob flows; snapshot metadata persisted and destination validation enforced. * **Documentation** * Complete etcd backup/restore README and admin/tenant scenario guides. * **Examples & Tests** * End-to-end demo scripts (helpers, stepper, cleanup, runner), Bats e2e, and extensive unit tests. * **Chores** * RBAC updates for etcd access, chart version bumped, demo env files added to .gitignore. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2641?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
d80f67e799 |
feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov DaniilandIvanHunters
|
14e196cf22 |
fix(postgres): drive postgresql.parameters int-or-string from the generator
Ivan's original schema edits were correct in intent but lived in the
generated artifacts (values.schema.json and the embedded openAPISchema
in postgres-rd/cozyrds/postgres.yaml). `make generate` reverts them
because values.yaml's `@field {map[string]string}` annotation is the
source of truth -- same regeneration trap that caught #2466 (issue #2597).
cozyvalues-gen v1.4.0 (cozystack/cozyvalues-gen#25, merged 2026-05-21)
adds an `intOrString` type alias for map values that emits
`additionalProperties.anyOf: [integer, string]` plus
`x-kubernetes-int-or-string: true`. Move the change to source by
flipping the annotation, then regenerate:
- packages/apps/postgres/values.yaml: `@field {map[string]intOrString}`
with updated description noting integer values are accepted.
- packages/apps/postgres/values.schema.json: regenerated -- now carries
anyOf + x-kubernetes-int-or-string: true.
- packages/system/postgres-rd/cozyrds/postgres.yaml: embedded
openAPISchema regenerated to match.
- packages/apps/postgres/README.md: regenerated type column.
- api/apps/v1alpha1/postgresql/types.go: Parameters is now
`map[string]intstr.IntOrString`.
The result survives future regenerations. The template's
`{{ \$value | toString | quote }}` already coerces both inputs to
quoted strings, verified end-to-end with helm template against both
`max_connections: 100` (int) and `max_connections: "100"` (string) --
identical render output. Dangerous-param block list still rejects
archive_cleanup_command and recovery_end_command additions from the
prior commit.
Co-Authored-By: IvanHunters <xorokhotnikov@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
IvanHunters
|
388e823916 |
fix(postgres): block archive_cleanup_command and recovery_end_command
Add archive_cleanup_command and recovery_end_command to the dangerous parameters blocklist in db.yaml. Both are PostgreSQL shell-exec hooks identical in threat model to archive_command/restore_command and must not be settable by tenants. Also update the BLOCKED list in values.schema.json, values.yaml, and the embedded openAPISchema in cozyrds/postgres.yaml to reflect the two new blocked parameters. Additionally update the embedded openAPISchema in cozyrds/postgres.yaml to allow integer values for postgresql.parameters (anyOf string|integer), matching the fix already applied to values.schema.json. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniil
|
ed0297eb87 |
refactor(build): standardize image tagging to fix concurrent PR push conflicts
Every package Makefile tagged images via `settag` which, on a non-release commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every image's `:latest` manifest in the OCIR registry. Two outliers (metallb, redis-operator) used hardcoded version tags and collided across PRs even when `settag` would not have. Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro driven by three env vars: - IMAGE_TAG — build-unique tag, always pushed - PUBLISH_VERSIONED — also push :<component-version> - PUBLISH_FLOATING — also push :latest PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_* on, preserving the previous release output and adding a uniform :<git-tag> tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0 already off. Image consumers reference digests pinned into values.yaml by each Makefile, so the tag is decorative for cluster-side pulls; the build-unique tag is sufficient for E2E. Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing in the repo or downstream values.yaml consumed it. The same information is preserved via the new uniform `:<git-tag>` plus the existing `:<component-version>` on release. ubuntu-container-disk builds a separate image per Kubernetes version in a loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop iterations within a single build do not push to the same tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
243d20ea30 |
fix(cluster-api): fix kamaji OOM and set limits on unset providers (#2708)
## What this PR does Fixes resource specs on CAPI provider pods in `cozy-cluster-api`. **Kamaji control-plane-provider — fix OOMKilled crashloop.** The override in `packages/system/capi-providers-cpprovider/templates/providers.yaml` targeted `containers[name=manager]`, but the upstream container in `docker.io/clastix/cluster-api-control-plane-provider-kamaji` is named `controller`. The CAPI operator merges container overrides by container name, so the intended `cpu: 1 / memory: 1024Mi` limits were silently dropped and the upstream default of `memory: 128Mi` was used. The kamaji controller pod therefore restarts repeatedly with `exit code 137` (OOMKilled), even at idle. **Core / bootstrap / kubevirt providers — add limits.** Upstream `*-components.yaml` for these three providers ships with no `resources:` block, and cozystack added no override. The pods run BestEffort and are first to evict under memory pressure. Set modest limits (`500m / 256Mi`) and requests (`10m / 64Mi`) matching typical CAPI controller-manager footprint. ### Release note ```release-note fix(cluster-api): fix the resource override for the kamaji control-plane-provider (was silently ignored due to a wrong container name, causing OOMKilled restarts) and set requests/limits for the core, kubeadm-bootstrap, and kubevirt-infrastructure providers (were running BestEffort). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated provider deployment configurations to establish resource management parameters (CPU/memory requests and limits) for container deployments across multiple provider types, improving scheduling optimization and resource predictability * Standardized container naming conventions in provider templates for consistency <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2708?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
6ae66f72a6 |
fix(cluster-api): fix kamaji OOM and set limits on unset providers
The kamaji control-plane-provider override in templates/providers.yaml targeted containers[name=manager], but the upstream container is named "controller". The CAPI operator merges container overrides by name, so the intended cpu: 1 / memory: 1024Mi limits were silently dropped and the upstream default of memory: 128Mi was used, producing repeated OOMKilled restarts of the kamaji controller (exit code 137). The other three providers (core, bootstrap, infraprovider/kubevirt) ship upstream with no resources block and have no cozystack override, so they run BestEffort and are first to evict under memory pressure. Set modest limits (500m / 256Mi) and requests (10m / 64Mi) matching typical CAPI controller-manager footprint. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei KvapilandClaude
|
4d5486cd84 |
chore(controller): remove dead dashboard controller
The dashboard controller used to project ApplicationDefinitions into a set of dashboard.cozystack.io CRDs (Sidebar, Factory, Navigation, Breadcrumb, CustomFormsOverride, CustomFormsPrefill, CustomColumnsOverride, MarketplacePanel, TableUriMapping, CFOMapping, BreadcrumbInside) consumed by the legacy UI. cozystack-ui now reads ApplicationDefinitions directly, so the controller, its API types, the generated CRDs and the related RBAC rules are no longer used. Removes: - internal/controller/dashboard - api/dashboard/v1alpha1 - packages/system/cozystack-controller/definitions/dashboard.cozystack.io_*.yaml - dashboard.cozystack.io wiring from cmd/cozystack-controller/main.go - dashboard.cozystack.io RBAC rules from cozystack-controller and cozy-dashboard charts - ./api/dashboard/... codegen path from hack/update-codegen.sh Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Myasnikov Daniil
|
e5437240e7 |
fix(opensearch-operator): use values.yaml override instead of patch
Move the kube-rbac-proxy image override from a patch + vendored-chart
edit to a values.yaml override under kubeRbacProxy.image.repository.
This matches the project convention documented in docs/agents/overview.md
("Override configuration in root values.yaml") and the pattern used by
sibling packages such as kamaji.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
d0a0b06afd |
fix(opensearch-operator): replace deprecated kube-rbac-proxy image
The gcr.io/kubebuilder/kube-rbac-proxy image is no longer available since GCR was deprecated. Replace it with quay.io/brancz/kube-rbac-proxy from the original upstream author. Applied as a local patch in patches/ because upstream chart 2.8.1+ drops kubeRbacProxy entirely but also bumps CRDs to controller-gen v0.19.0 with significant schema changes — out of scope for this fix. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey Kolkov
|
15c015c226 |
fix(platform): fix broken dashboard login by token
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
cozystack-ci[bot]
|
9f655e29ff |
Prepare release v1.4.0-rc.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
680803acae |
feat(dashboard): replace UI with new cozystack-ui build (#2507)
## Summary Replaces the old openapi-ui with the new cozystack-ui build from the cozystack-ui repository. ## Key Changes ### New UI - **Image**: `999669/cozystack-ui:latest@sha256:ea4e832c2...` - **Architecture**: `linux/amd64` - **Repository**: https://github.com/cozystack/cozystack-ui (feat/ui-overhaul branch) ### Removed Components - Removed BFF (Backend-For-Frontend) container - new UI communicates directly with Kubernetes API - Removed old openapi-ui and openapi-ui-k8s-bff build targets from Makefile - Removed old UI Dockerfiles and patch files ### RBAC Updates - Added read permissions for `cozystack.io/applicationdefinitions` to fix 403 errors - New UI requires access to applicationdefinitions for marketplace and catalog functionality ### Bug Fixes - Fixed VNC WebSocket connections to use dynamic URLs instead of hardcoded localhost:8001 - VNC now works properly in-cluster deployments ## Testing Tested on dev3.infra.aenix.org cluster: - ✅ Dashboard loads correctly - ✅ Application definitions accessible - ✅ VNC connections working - ✅ Backup pages functional - ✅ All TypeScript errors resolved ## Migration Notes The new UI is a complete rewrite with: - Modern React 19 + TypeScript stack - Improved backup management pages - Better type safety and error handling - Direct Kubernetes API communication (no BFF needed) |
||
|
Andrey Kolkov
|
c5b46f31fe |
fix(platform): align field of Altinity backup-strategy for similarity… (#2645)
… with others <!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note align Altinity backup strategy field name for similarity with others ``` |
||
|
myasnikovdaniil
|
42b5cc4fd6 |
Merge branch 'main' into feat/new-ui
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
Andrey Kolkov
|
84874353ce |
fix(platform): align field of Altinity backup-strategy for similarity with others
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
cozystack-ci[bot]
|
252c18f3e8 |
Prepare release v1.4.0-rc.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Andrey Kolkov
|
92930ad1aa |
feat(platform): add implementation of backup-strategy controller for foundationdb
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
5f83ae72a8 |
feat(platform): implement backup strategy for mariadb (#2605)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note implement backup strategy for MariaDB ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added native MariaDB backup/restore strategy using the operator-native BackupClass/Plan/RestoreJob flow with example manifests for end-to-end workflows. * **Documentation** * Added MariaDB how‑to and examples; marked legacy restic/chart-side backup parameters as DEPRECATED and documented migration guidance. * **Tests** * Added unit tests covering MariaDB backup/restore reconciliation, templating, storage handling, idempotency, URIs, and deadlines. * **Chores** * Extended controller RBAC to allow observing MariaDB operator resources. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2605) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
f38a047aa6 |
feat(kilo): add patched image build with cross mesh granularity (#2604)
## What this PR does Adds infrastructure for building a patched kilo image from upstream `main` branch source, following the same pattern as the kamaji package. The patch introduces a new `cross` mesh granularity mode — WireGuard tunnels are created only between nodes in different locations, while intra-location traffic stays on the CNI overlay. Changes: - `images/kilo/Dockerfile` — multi-stage build: fetches upstream tarball, applies patches, builds `kg`/`kgctl` binaries with CNI plugins and iptables-wrapper - `images/kilo/patches/feat-add-cross-mesh-granularity.diff` — adds `CrossGranularity` to the kilo binary - `Makefile` — new `image:` target for building the patched image, updated `update:` target to pin upstream commit SHA ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(kilo): add patched image build with cross mesh granularity support ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added "cross" mesh granularity mode for finer topology grouping, adjusted routing/encryption behavior * Added e2e tests covering cross-mode scenarios for improved reliability * **Chores** * Build automation now tracks latest main commit and derives image digests from build metadata * Introduced multi-stage container build and runtime packaging for kilo, including required networking tooling * **Configuration** * Updated default image reference to a new repository and digest [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2604) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
92d128e8ff |
feat(keycloak): support extraEnv values and userProfile customization (#2613)
## What this PR does Two related quality-of-life additions to the Keycloak system component: 1. **`keycloak` chart now accepts `extraEnv`.** Operators can append extra `EnvVar` entries to the Keycloak StatefulSet container without forking the chart. Useful when a custom theme needs runtime configuration via `kcContext.properties`, or when an extension expects environment variables. 2. **`keycloak-configure` chart now exposes a top-level `userProfile` value** that is mirrored verbatim into `ClusterKeycloakRealm.spec.userProfileConfig`. This lets operators declare custom user-profile attributes (required fields, validators, permissions, ...) declaratively, alongside the existing `login` and `smtp` blocks. Both values default to empty, so existing deployments are unaffected. ### Release note ```release-note feat(keycloak): support extraEnv values and userProfile customization ``` |
||
|
Andrey Kolkov
|
62605a198b |
feat(platform): implement backup strategy for mariadb
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Kirill Ilin
|
3e5b370f74 |
chore(keycloak): use generic example values in chart comments
Replace deployment-specific examples in the new extraEnv and userProfile docs with generic placeholders so the chart comments do not lean on any particular downstream use case. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
2261ef7e25 |
feat(keycloak-configure): plumb userProfile into ClusterKeycloakRealm
Operators can now declare custom user-profile attributes (required fields, validators, permissions, ...) via a new top-level userProfile value. The value is mirrored verbatim into the ClusterKeycloakRealm.spec.userProfileConfig managed by the EDP keycloak-operator. The default remains empty, so realms that don't need custom user profile configuration are unaffected. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
6413052abc |
feat(keycloak): expose extraEnv values to the StatefulSet
Custom themes (Keycloakify, FTL) often need runtime values from the deployment environment via kcContext.properties. Adding an extraEnv list to the chart lets operators inject those values per cluster without forking the chart. The new field appends verbatim to the container env list and accepts the full corev1.EnvVar schema, including valueFrom for secret or configMap-backed values. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Aleksei SviridkinandMatthieu ROBIN
|
fdffedbd4b |
feat(apps): instance-type resourcesPreset across apps and extras
Rewrite resourcesPreset values in every apps/* and extras/* chart from the legacy flat names to the new instance-type form, using the 1:1 mapping: nano -> t1.nano micro -> t1.micro small -> t1.small medium -> c1.small large -> c1.medium xlarge -> c1.large 2xlarge -> c1.xlarge Each affected values.yaml carries the canonical 47-value enum (40 instance-type names plus 7 legacy aliases) so the schema accepts both forms during the deprecation window. cozyvalues-gen regenerates the matching api/apps/v1alpha1/*/types.go kubebuilder enum and defaults, the chart values.schema.json, the README parameter table, and the ApplicationDefinition openAPISchema in packages/system/*-rd/cozyrds. Three hardcoded legacy literals in chart templates migrated: - packages/apps/harbor/templates/harbor.yaml:121 (portal) - packages/apps/opensearch/templates/opensearch.yaml:43 (dashboards) - packages/system/keycloak/templates/sts.yaml:115 (heap fallback) packages/apps/foundationdb/Makefile gains the missing hack/update-crd.sh call. Without it, every regen left foundationdb-rd cozyrds stale and new FoundationDB app CRs through cozystack-api were rejected at validation for any non-legacy resourcesPreset value. Every default-rendered chart produces identical CPU and memory to the value applied before the rename, courtesy of the legacy alias table in cozy-lib and the 1:1 default mapping. Co-authored-by: Matthieu ROBIN <matthieu@hidora.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Arsolitt
|
0be25ce59e |
build(kilo): pin patched image to cozystack registry
Point values.yaml to ghcr.io/cozystack/cozystack/kilo with the freshly built image digest, replacing the upstream ghcr.io/squat/kilo reference. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
e9c3817f4f |
fix(kilo): remove stale Alpine repo override in runtime stage
Address review feedback from gemini-code-assist on Dockerfile:36: Remove custom Fastly CDN repo pinned to v3.20 while base image is alpine:3.23 — use default repos from the base image instead. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Andrey Kolkov
|
250e1b3376 |
feat(platform): implement backup strategy for clickhouse application
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Arsolitt
|
269fe97e34 |
feat(kilo): add patched image build infrastructure
Add Dockerfile for building kilo from upstream main branch source with patch support, CNI plugins, and iptables-wrapper. Update Makefile with `image:` target for building the patched image and an updated `update:` target that pins the upstream commit SHA. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
6f83d9a794 |
feat(ouroboros): bump to v0.7.2 with debug logging on tenants
The v0.7.2 ouroboros release adds --log-level / OUROBOROS_CONTROLLER_LOG_LEVEL, plants Debug logs at every informer event handler (AddFunc/UpdateFunc/ DeleteFunc with kind+namespace/name via describeObj) and at reconcile entry/exit, wires SetWatchErrorHandler on Ingress / Gateway / HTTPRoute informers, and surfaces cache-sync elapsed at Info. The diagnostic surface targets the kamaji-tenant-via-konnectivity flake where bats sees a fresh Ingress create succeed against the tenant apiserver but the controller's coredns-custom rewrite lands roughly 7-10 minutes later — without per-event visibility the failure mode is indistinguishable between 'watch never fires', 'reconcile silently errors', and 'API list returns empty until reflector reconnect'. Bumps the chart-manifest digest, the controller-image digest, and the cozystack-side image-pin assertion in lockstep. The bats deadline for the coredns-custom rewrite check rises from 180s to 900s — even on the broken-watch path the reflector eventually reconnects within that budget, so the test stops failing on the symptom while we keep the diagnostic on to learn what the symptom actually is. cozystack tenants force controller.logLevel: debug via the defaultOuroborosValues helper; operators that want quieter logs can override through addons.ouroboros.valuesOverride. Two helm-unittest fixtures pin both the default and the override path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
45e21a330f |
feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561)
## What this PR does Adds [`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0) — a Go reimplementation of [`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy) — and wires it into cozystack as a *conditional default*: a single `publishing.proxyProtocol: true` switch turns PROXY-protocol on at the host ingress-nginx **and** auto-deploys ouroboros to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arriving at ingress-nginx without the PROXY header it now requires). Default stays `false` — clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. The chart is platform-agnostic at v0.7.0: the controller and proxy binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf` inside the pod and compose the rewrite target FQDN and the backend host at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on `cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`, federations on `k8s.example.com`, custom-domain RKE2/k3s — all work without operator override. Verified end-to-end on a `cozy.local` cluster: controller logs `cluster-domain=cozy.local` at startup, chart emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked `--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy ClusterIP. Operators that prefer a chart-baked FQDN can still set `controller.clusterDomain` explicitly — the explicit-override path is preserved for non-Service backends and for split-horizon DNS topologies. The chart runs in `coredns` mode for the host (the host CoreDNS is Talos-managed and does not carry an `import` directive, so the alternative would silently no-op there) and in `coredns-import` mode for tenant deployments (tenants run cozystack-coredns, which carries the import directive and ships an empty `kube-system/coredns-custom` ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the `ouroboros.override` data key without ever touching the chart-rendered Corefile, so it does not race the chart that re-templates `data.Corefile` on every Flux reconcile). ### Layout * `packages/system/ouroboros/` — vendored chart from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`: `controller.mode=coredns` (host scenario), `proxy.target.namespace=cozy-ingress-nginx` (override the chart-default `ingress-nginx` namespace; chart-default `serviceName=ingress-nginx-controller` matches as-is), Gateway-API watching left at upstream's default of `false`. `proxy.target.host` is left empty so the proxy composes the backend host at runtime. Image is digest-pinned at `0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c` and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is intentionally not mirrored under `ghcr.io/cozystack/*`. The chart manifest itself is also digest-pinned via `OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e` in the package Makefile, with a Chart.yaml-version vs. Makefile-version assert that aborts the update target on lockstep drift. Helm-unittest in `tests/` covers controller args, RBAC narrowing per mode, Gateway-API gating, runtime-composition flag emission in both `coredns` and `coredns-import` modes, image pin drift. * `packages/system/coredns/templates/coredns-custom.yaml` + `values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed `import` prerequisite, plus the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts` mount at `/etc/coredns/custom/`. Same package serves host and tenant CoreDNS, so this covers both layers without duplication. Helm-unittest pins the rendered Corefile via a snapshot so future `make update` does not silently drop the import directive. The empty rendered ConfigMap deliberately omits `data:` — Helm three-way merge has no template-side keys to compare against, so apiserver-side runtime writes (the `ouroboros.override` data key) survive every chart upgrade. The `notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the invariant. * `packages/core/platform/sources/ouroboros.yaml` — new `cozystack.ouroboros` PackageSource for the host install (`dependsOn: cozystack.networking, cozystack.ingress-application`). * `packages/core/platform/sources/kubernetes-application.yaml` — `kubernetes-ouroboros` component registered against `system/ouroboros` so the tenant HelmRelease can reference it. * `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` — tenant-side HelmRelease deployed via the tenant admin-kubeconfig, mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues` helper flips `mode` back to `coredns-import` for tenants and mirrors per-tenant `addons.gatewayAPI.enabled` into the controller (forcing `--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the controller on `WaitForCacheSync`). `controller.clusterDomain` is intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from `/etc/resolv.conf` and composes the rewrite target at runtime, so the same wrapper renders correctly on `cluster.local`, `cozy.local`, or any custom tenant cluster-domain. Operators on a non-default cluster-domain can still override via `valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a conditional `gateway-api-crds` entry on the same gate. If `addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled` is false, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool `true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML `yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via `toString`-then-`lower`-then-string-compare; direct bool `eq` would error on type mismatch when an operator passes the value as a string from `--set`, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. Tenant disable is single-step: flipping `addons.ouroboros.enabled: false` deletes the rendered HelmRelease, helm-controller runs `helm uninstall` on the tenant, and the chart's pre-delete cleanup hook (`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the `ouroboros.override` key on the tenant's `kube-system/coredns-custom` ConfigMap automatically. No render-time guard is wired on the tenant side — gating on a `lookup` for the leftover HelmRelease would deadlock the routine disable (the parent render runs before helm-controller applies the missing-child diff, so the lookup would always find the leftover at the moment of the flag flip). If the chart's hook fails to land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job timeout), operators recover manually via the documented tenant cleanup recipe. * `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef with `enabled` + `valuesOverride`. `make generate` re-emitted README, JSON schema, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. * `packages/core/platform/values.yaml` — new `publishing.proxyProtocol` flag with extensive comment covering the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on, otherwise external traffic breaks), the host-vs-tenant fan-out (the flag does not cascade into tenants), the Talos re-render flap window, and the asymmetric disable path. The host disable path: flipping back to `false` reverts ingress-nginx PROXY-protocol config (flip the LB off first), but the host ouroboros itself stays installed because its Package CR carries `helm.sh/resource-policy: keep` — an explicit `kubectl delete package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages` entry) actually triggers helm uninstall, at which point the chart's pre-delete cleanup hook fires and patches the host Corefile automatically. The values comment ships a manual `sed`-based cleanup recipe as a fallback for the rare case where the chart's hook fails to land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off a `lookup` against the leftover Package CR to catch operators who flip the flag back without confirming the cleanup landed. * `packages/core/platform/templates/bundles/system.yaml` — flips between the default-form and explicit-form `cozystack.ingress-application` Package emission so the host ingress-nginx gets `use-proxy-protocol` + `real-ip-header=proxy_protocol` injected only when the flag is true (`use-forwarded-headers` and `compute-full-forwarded-for` are deliberately *not* on the list — they would let any upstream proxy spoof `X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`. Host Gateway-API watching is left at the upstream chart's default of `false` because cozystack does not install Gateway-API CRDs cluster-wide on the host. * `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in tenant, controller pod running, and after creating a TLS Ingress the rewrite snippet shows up in the import ConfigMap; an in-tenant `dig` from a netshoot pod also confirms CoreDNS actually serves the rewrite — pod-readiness alone passes vacuously since the upstream chart ships no readiness probes). ### Why not KEP-1860 `ipMode=Proxy` Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB frontend entirely when `ipMode != VIP`, breaking Service ingress for L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters fronted by an external CCM-managed proxy LB. ouroboros is the correct fix for L2/BGP-announce topologies that cozystack ships by default. ### Coverage Covered by helm-unittest (host chart, coredns wrapper, tenant addon — including the `kubernetesProvider` mock that exercises the lookup-driven `proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil CI/dry-run path, the gateway-api truthy-spelling acceptance across `true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that `--proxy-service-name` + `--proxy-service-namespace` are emitted instead of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import` modes, the assertion that `--target-service-name` + `--target-service-namespace` are emitted for the proxy backend, and the `_namespace.etcd`-empty render gate) and a tenant e2e bats test that creates a real Ingress, asserts the rewrite snippet lands in the import ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry loop tolerating CoreDNS reload latency). The host-level `publishing.proxyProtocol: true` path is not e2e-tested in this PR — it requires a PROXY-protocol-aware upstream LB, which the cozystack e2e harness does not stand up. That path is exercised manually and by docs review. End-to-end runtime auto-detect was verified manually on a `cozy.local` cluster (dev17): chart deployed at v0.7.0, controller logged `cluster-domain=cozy.local`, the rewrite block landed in `kube-system/coredns` Corefile pointing at `test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig <ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first attempt, and the chart's pre-delete hook restored the Corefile cleanly on `helm uninstall`. ### Supply-chain note Image and chart are pulled from a contributor's personal namespace (`ghcr.io/lexfrei/...`). Existing precedent (`packages/system/etcd-operator/Makefile` pulls from `ghcr.io/aenix-io/...`) is similar but `aenix-io` is the cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer call. Pairs with cozystack/website#527 (operator docs). ### Release note ```release-note feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with configurable enablement and tenant overrides. * Added a packaged Ouroboros chart providing proxy, controller, CoreDNS integration, external‑DNS and etc‑hosts modes. * Introduced PROXY‑protocol gating with an explicit acknowledgement flag. * **Documentation** * Expanded addon reference and platform docs with detailed Ouroboros/PROXY‑protocol configuration and defaults. * **Tests** * Added extensive unit and e2e tests covering rendering, RBAC, uninstall cleanup, and DNS behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
8da1e7a40a |
feat(platform): implement backup strategy for postgres (#2552)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note Implemented CNPG-strategy controller for postgres backups with supporting in-place and to-copy restores. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * CNPG-based backup strategy for PostgreSQL with S3-backed object store and controller support * Option to reference existing Kubernetes Secret for S3 credentials (configurable key names) * Controller supports scheduled/ad-hoc backups and both in-place and to-copy restores; safer credential handling (no inline keys) * **Examples** * Complete manifests: bucket, credentials, CNPG strategy, BackupClass, Plan, BackupJob, RestoreJob, Postgres source/target * **Security/RBAC** * Controller RBAC expanded for CNPG resources; tests enforce no cluster-wide Secret read * **Documentation** * README, chart values/schema and UI schema updated to document secret-reference workflow and restore guidance * **Tests** * New end-to-end and unit tests covering CNPG flows, restore semantics, cleanup behavior and RBAC constraints <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
653519a472 |
fix(coredns): sanitize Chart.Version build metadata in coredns-custom label
The cozystack build pipeline tags chart versions as `0.0.0+<git-sha>`. The new coredns-custom ConfigMap interpolated `.Chart.Version` directly into the `helm.sh/chart` label, producing `cozy-coredns-0.0.0+<sha>` which the apiserver rejects: label values follow RFC 1123 and `+` is not a legal character. The chart never installed, kube-system/coredns inside tenant clusters never came up, every dependent HelmRelease stalled, and the e2e suite hit the 2h job timeout without reaching the bulk of the bats tests. Apply the standard `replace "+" "_" | trunc 63 | trimSuffix "-"` sanitization, matching the pattern the vendored upstream coredns chart already uses in its own `helm.sh/chart` helper a few directories down. Add a helm-unittest case that overrides the chart version to a build- metadata form (`0.0.0+abc1234567`) and asserts the rendered label both matches the sanitized literal and conforms to the RFC 1123 label-value regex. Without the version override the bug is invisible to the test suite — Chart.yaml ships the bare `0.0.0` placeholder and helm-unittest defaults to that, so any future template that interpolates `.Chart.Version` into a label value cannot regress past CI silently. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrey Kolkov
|
ed5ae16e36 |
feat(platform): implement backup strategy for postgres application
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Aleksei Sviridkin
|
a9c1298b42 |
[gpu-operator] Update to v26.3.1 and add experimental vGPU variant (#2323)
## What this PR does Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an **experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and newer GPUs. Changes: - Update vendored upstream chart to v26.3.1 (CDI enabled by default, component version bumps). - Pin chart version in Makefile for reproducibility. - Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload: vm-vgpu` so the chart activates the vGPU Manager DaemonSet without per-node labelling. - Add `vgpu` variant to PackageSource alongside the existing `default` (passthrough). - Rename `values-talos.yaml` → `values-passthrough.yaml` and pin `defaultWorkload: vm-passthrough` for symmetry with the vgpu variant. The overrides apply to any host OS using vfio-pci, not Talos specifically. - Refresh `hack/dcgm-default-counters.csv` to match the new dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED` and other fields introduced upstream. `hack/check-gpu-recording-rules.bats` passes against the new fixture. - Add `hack/check-gpu-operator-variants.bats` that renders both variants via `helm template` and asserts the pinned ClusterPolicy values, so the next chart bump cannot silently un-pin them. - Add internal documentation for the vGPU bring-up. The `vgpu` variant assumes the user supplies a proprietary vGPU Manager image via Package CR component values — the .run cannot be redistributed. ### `vgpu` variant is experimental NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no parent devices found" and is disabled by default. Until an SR-IOV-aware controller is shipped, profile assignment is an out-of-band step (`echo <id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after every node reboot. Without it, `permittedHostDevices.pciHostDevices` reports zero allocatable resources. The variant ships so the chart bump and the underlying skeleton (DaemonSet + image plumbing + docs) land together; production deployments need an automated profile-assignment mechanism in addition to this PR. ### KubeVirt requirement KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU: SRIOV support", merged 2026-04-10) which is `main`-only — released tags `v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted at the next minor KubeVirt release (v1.9.0); track the PR for the actual release tag. Backports to released branches are not planned. ### Behavior changes inherited from upstream chart v25 → v26 The upstream chart flipped two defaults that are pinned back to `false` here, so `default` (passthrough) deployments do not silently change footprint on upgrade: - `ccManager.enabled` was flipped to `true` upstream — would auto-enable Confidential Computing on Hopper hardware (H100/H200/B100) after upgrade. - `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet would render even in passthrough mode. Both stay opt-in via Package CR overrides. The new bats test guards the pin. Other inherited changes that are user-visible and worth pinning down before upgrade: - `driver.licensingConfig.configMapName` is deprecated upstream in favour of `driver.licensingConfig.secretName`. The deprecation marker lives in the CRD's `description` field (visible via `kubectl explain`) — there is no helm-render-time warning, the chart template silently prefers `secretName` and falls back to `configMapName`. Operators with existing licensing overrides should migrate the key on the next chart upgrade. - `driver.version` upstream default jumped from `570.x` to `580.x`. Invisible on the `default` variant (which sets `driver.enabled: false`), user-visible for anyone running the upstream Ubuntu driver path. - `vfioManager.image` switched from `nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to `nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every `default` (passthrough) deployment on upgrade — different image, different layer, different image-pull surface. Worth pre-mirroring for air-gapped clusters. - `dcgmExporter.image` switched to the `-distroless` flavour. Smaller surface, fewer base-image CVEs, but no shell — operators who used to `kubectl exec` into the exporter for ad-hoc debugging will need to switch to `kubectl debug` or container ephemeral images. - `CDI` (Container Device Interface) is enabled by default in v26.3.1; this is upstream's recommended setting and is left as-is. - `node-feature-discovery` subchart source switched from the GitHub Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped installations need to mirror the new OCI URL. ### Talos Talos is **not recommended** for the vGPU variant. NVIDIA does not publicly distribute the vGPU guest driver — it requires NVIDIA Enterprise Portal access — and Sidero closed siderolabs/extensions#461 noting that they cannot support vGPU "unless NVIDIA changes their licensing terms or provides us a way to obtain, test, and distribute the software". Building a Talos system extension that bakes in the vGPU driver is therefore not feasible without a private fork that violates the EULA. Passthrough (`default` variant) on Talos is unaffected. ### Release note ```release-note [gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced vGPU support variant with end-to-end configuration guide, validation procedures, and troubleshooting documentation * **Updates** * Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3 * Improved component health checks and added PodDisruptionBudget availability controls <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
8e78536cc3 |
fix(cert-manager): bump chart to v1.20.2 (#2562)
## What this PR does Bumps the vendored cert-manager chart from v1.19.3 to v1.20.2. Pure upstream bump via `make update` — no cozystack-side template changes. ### Why [Commit `2b6e20cc`](https://github.com/cozystack/cozystack/commit/2b6e20cc) migrated the platform's ACME HTTP-01 setup to the modern `ingressClassName` API on both the ClusterIssuer's solver and the per-tenant override annotation `acme.cert-manager.io/http01-ingress-ingressclassname`. The override annotation, however, was only added to upstream cert-manager in v1.20.0 ([cert-manager#8244]) — v1.19.x silently ignores it. Result: cert-manager always falls back to the ClusterIssuer's solver class (`tenant-root`), so HTTP-01 solver Ingresses for non-`tenant-root` tenants land on the wrong controller and Let's Encrypt validation fails for tenant-scoped Ingresses (grafana, harbor, alerta, etc. inside child tenants). Targeting v1.20.2 (not .0): - [cert-manager#8655] (in v1.20.1) — restores `clusterissuers/issuers/finalizers` RBAC dropped in v1.20.0. - [cert-manager#8665] (in v1.20.2) — webhook helm-template fix. ### Verified live on dev10 | Stage | `Challenge.spec.solver.http01.ingress.ingressClassName` | |---|---| | Pre-bump (v1.19.3) | `"tenant-root"` (bug) | | Post-bump (v1.20.2) | `"<tenant>"` (fixed) | Full propagation chain confirmed end-to-end: source Ingress annotation -> ingress-shim sets `http01-override-ingress-ingressclassname` on the Certificate -> acmeorders sets `spec.solver.http01.ingress.ingressClassName` on the Challenge -> http01 solver creates the solver Ingress on the tenant's class. ### Upgrade considerations - **Container UID/GID change**: 1000/0 -> 65532/65532 across controller, webhook, cainjector. PSP / SecurityContext rules pinning UID 1000 will need updating. - **Helm chart image structure** refactored from `image.repository` to `imageRegistry` + `imageNamespace` + `image.name`. Cozystack's wrapper `values.yaml` is empty, so rendered image references are unchanged (`quay.io/jetstack/cert-manager-{controller,webhook,cainjector,acmesolver}:v1.20.2`). - **NetworkPolicy templates restructured**: legacy `networkpolicy-egress.yaml` removed; per-component `networkpolicy-cainjector.yaml` and `networkpolicy-cert-manager.yaml` added. All gated on `*.networkPolicy.enabled: false` (default). No behavioral change. - **CRDs**: `selectableFields` added (Kubernetes >=1.30) and Azure DNS01 `zoneType` enum. Both pure additions. - **RBAC**: gains `clusterissuers/issuers/finalizers` and `gateway.networking.k8s.io.listenersets[/finalizers]`. - `DefaultPrivateKeyRotationPolicyAlways` feature gate moves Beta -> GA, can no longer be disabled. Already default since v1.18; no behavior change. ### Release note ```release-note fix(cert-manager): bump to v1.20.2 - fixes HTTP-01 solver IngressClass propagation for tenant-scoped Ingresses (the `acme.cert-manager.io/http01-ingress-ingressclassname` annotation was a no-op in prior cert-manager versions). Default container UID/GID changes from 1000/0 to 65532/65532 - review your PSP/SecurityContext rules. ``` [cert-manager#8244]: https://github.com/cert-manager/cert-manager/pull/8244 [cert-manager#8655]: https://github.com/cert-manager/cert-manager/pull/8655 [cert-manager#8665]: https://github.com/cert-manager/cert-manager/pull/8665 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Added network policy support for controller, webhook, and CA injector * Added Azure DNS zone type selection (public/private) * Added webhook client verification configuration options * Extended pod disruption budget configuration with unhealthy pod eviction policy * **Updates** * Upgraded cert-manager to v1.20.2 * Enhanced global image registry and namespace configuration * Expanded issuer reference field selectors * Added support for extra containers in controller pod * Updated issuer terminology and configuration * **Documentation** * Updated Helm chart documentation with new configuration options * Updated post-installation guidance <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7f01c7eb32 |
feat(platform): single switch for host PROXY-protocol + ouroboros
Adds publishing.proxyProtocol on the platform values: a single flag that both turns PROXY-protocol on at the host ingress-nginx (via the ingress-application Package CR's component values dict) and emits the new cozystack.ouroboros Package CR for the host install. The two pieces are coupled by design — turning PROXY-protocol on at ingress-nginx without ouroboros breaks intra-cluster traffic to the cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits the LB IP, the connection arrives at ingress-nginx without the PROXY header it now requires, and ingress-nginx closes the connection). The new sources/ouroboros.yaml PackageSource declares the host chart with dependsOn: cozystack.networking + cozystack.ingress-application so install ordering is deterministic. The bundle template emits the host ouroboros Package via the standard helper — the parent chart's Gateway-API watching default of false stays in place, because cozystack does not install Gateway-API CRDs cluster-wide on the host (system/ gateway-api-crds is referenced only from the kubernetes-application PackageSource, tenant scope). Forcing --gateway-api on a host without the CRDs would CrashLoop the controller on WaitForCacheSync against missing GVRs. The flag default is false. Clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. Disabling: flipping publishing.proxyProtocol back to false stops emitting the cozystack.ouroboros Package CR, but every Package CR carries helm.sh/resource-policy: keep — Helm leaves the existing Package on the cluster so ouroboros stays installed and the live Corefile rewrite block keeps pointing at the still-running ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To actually uninstall ouroboros the operator removes the Package CR explicitly (or adds the package name to bundles.disabledPackages) and runs the documented Corefile cleanup recipe in the same window — otherwise stale rewrites point at a deleted Service and in-cluster DNS for hairpinned hostnames returns connection-refused. The values comment ships the cleanup recipe (a JSON merge-patch on data.Corefile that strips the BEGIN/END block while preserving labels, annotations, and other data keys including any Talos-managed metadata) and documents the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on), the host-vs-tenant fan-out (the flag does not cascade into tenant clusters; tenants opt in per-Kubernetes-CR via addons.ouroboros.enabled), and the Talos re-render flap window (machine-config pushes wipe the BEGIN/END markers; the controller re-applies on next reconcile, ≤10 minutes by default). Also drops the stale "distro-full" entry from the bundles.system.variant options comment in core/platform/values.yaml — the variant was removed in v1.0.1 and was never restored. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7ab940f309 |
feat(kubernetes): expose ouroboros as a per-tenant addon
Adds the OuroborosAddon typedef, addons.ouroboros knob, and tenant HelmRelease that deploys the cozy-ouroboros chart inside a tenant Kubernetes cluster via the host's flux-helm-controller and the tenant admin-kubeconfig — same shape as the cert-manager addon. The HelmRelease overrides controller.mode back to coredns-import because tenants run the cozystack-coredns wrapper (which carries the `import /etc/coredns/custom/*.override` directive); the parent chart's default of mode=coredns is the right shape only for the Talos-managed host CoreDNS. controller.gatewayApi.enabled is mirrored from the per-tenant addons.gatewayAPI.enabled — forcing --gateway-api on a tenant without Gateway-API CRDs CrashLoops the controller on WaitForCacheSync against missing GVRs. dependsOn grows a conditional gateway-api-crds entry on the same gate. controller.clusterDomain is intentionally NOT pre-set by the wrapper. The chart auto-detects it inside the controller pod by parsing /etc/resolv.conf — the search list of which is `*.<ns>.svc.<domain>` and reflects exactly the zone the tenant CoreDNS actually serves (governed by KamajiControlPlane.spec.networkProfile.clusterDomain defaulting to `cluster.local`, and the cozystack-coredns wrapper kubernetes-plugin zone, also `cluster.local`). Pre-setting from a host-side fact like `_cluster.cluster-domain` would override the auto-detect with a value that does not match what the tenant kube-dns serves, silently emitting NXDOMAIN-bound rewrites. Operators that run a tenant on a non-default cluster-domain can still override via addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain. If addons.ouroboros.enabled is true but ingressNginx is off, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool true, string "true"/"True"/"TRUE", numeric 1, YAML "yes"/"Yes"/"YES"/"on"/"On"/"ON"/"y"/"Y") via toString-then-lower-then-string-compare; direct bool eq would error on type mismatch when an operator passes the value as a string from --set, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. The acknowledgeUncleanRemoval gate is defence-in-depth for the rare edge case where helm uninstall fails to land cleanly (tenant unreachable, hook timeout, manual kubectl delete bypassing helm) — in the normal path the upstream chart's pre-delete cleanup hook (coredns-cleanup-hook.yaml, vendored at 0.6.0) nulls the ouroboros.override key automatically when the HelmRelease is uninstalled. The gate only blocks render when a leftover HelmRelease on the host kube-apiserver shows the previous install is still sitting on the cluster. The component is also registered in the kubernetes-application PackageSource (as kubernetes-ouroboros at path system/ouroboros) so the tenant HelmRelease's ExternalArtifact resolves. make generate re-emitted README.md, values.schema.json, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. helm-unittest covers: render gating on enabled flag, hard-fail when ingress-nginx is off, namespaces and labels, chartRef, dependsOn cilium/coredns/ingress-nginx + conditional gateway-api-crds, default coredns-import mode, valuesOverride precedence, gateway-api flag mirroring in both directions, gateway-api truthy-spelling acceptance (including case-insensitive on/On/True/TRUE), the lookup-driven acknowledge gate (kubernetesProvider mock asserts the failedTemplate path when a prior HelmRelease still exists on the host), the lookup-returns-nil CI/dry-run path (render proceeds when the cluster mock has no objects), the _namespace.etcd-empty render gate, and that controller.clusterDomain is NOT pre-set in the rendered HelmRelease (so chart auto-detect runs). Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9f8e7f9d4a |
feat(coredns): import /etc/coredns/custom into the Corefile
Extends the cozystack-coredns wrapper with the standard CoreDNS `import /etc/coredns/custom/*.override` directive in the .:53 server block and renders an empty kube-system/coredns-custom ConfigMap as a chart-managed prerequisite, mounted into the CoreDNS Deployment at /etc/coredns/custom/ via extraVolumes / extraVolumeMounts. The ConfigMap carries helm.sh/resource-policy: keep so the chart-managed empty value cannot strip what later writers add into it. This exposes a stable extension point on the cozystack CoreDNS for operators (or other cozystack components) that need to add plugin-only rewrite snippets without touching the chart-rendered Corefile — otherwise the chart would re-template data.Corefile on every Flux reconcile and overwrite the additions. The same package serves both host and tenant CoreDNS deployments, so this plumbing covers both layers without per-layer duplication. The volume mount is `optional: true` for defence-in-depth: a future chart upgrade that drops the templates/coredns-custom.yaml render still leaves CoreDNS healthy with no override snippets in play. helm-unittest fixtures pin the ConfigMap shape (name, namespace, keep annotation, no data), the `import` directive in the rendered Corefile, the presence of the `reload` plugin (required for live ConfigMap updates), and the volume + volumeMount on the CoreDNS Deployment. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
