PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Option providers RBAC block granted read on
objectstorage.k8s.io/bucketclasses, but no provider in the registry lists
BucketClasses and no schema field references a bucketclass source — the
bucket app's storage pool field uses the storagepool provider, which reads
seaweedfs HelmReleases. Drop the unused grant and pin its absence in the
chart test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The plan and backup option providers list namespaced plans and backups in
the backups.cozystack.io group, but the cozystack-api ClusterRole only
granted backupclasses there. The apiserver's privileged client runs as this
ServiceAccount, so both providers got forbidden — List silently dropped the
sources and Get returned 500, leaving the plan/backup dropdowns in the
backup forms empty.
Add plans and backups to the backups.cozystack.io rule and cover the grant
with a chart test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Introduce a read-only, namespaced virtual resource
core.cozystack.io/v1alpha1 Option that serves named lists of dropdown
options for the dashboard. Each object (one per source: gpu,
instancetype, instanceprofile, network, image, storagepool,
storageclass, vmdisk, backupclass) is computed on read by an in-process
provider using a privileged dynamic client, so tenants need no direct
access to Nodes, the KubeVirt CR, cluster instancetypes, etc.
Tenants get read access via the aggregated cozy:tenant roles; the
apiserver ServiceAccount gains read on the underlying sources.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since 23e399bd, but `Create` and `Delete` accepted the
parameter and never called it — admission was silently bypassed for both
verbs on every `apps.cozystack.io` resource (Tenant, MariaDB, Postgres,
Kubernetes, …).
This PR wires the missing `createValidation` and `deleteValidation`
calls into Create and Delete. Every ValidatingAdmissionPolicy and
ValidatingWebhook against `apps.cozystack.io/*` now fires on all three
verbs as the API contract requires. Per KEP-3488 aggregated APIServers
are responsible for enforcing admission on their own kinds;
cozystack-api is configured to do so via
`genericoptions.NewRecommendedOptions`, the only thing missing was the
explicit `*Validation` callbacks in the custom REST.
## Security — five guards, grouped by what they defend
Tenants in Cozystack interact with the platform exclusively through
`apps.cozystack.io/*` resources (Tenant, Bucket, Kubernetes, …) served
by `cozystack-api`. Tenant RBAC (`cozy:tenant:*` aggregated to a
RoleBinding in the tenant's own namespace) does not grant write access
to `gateway.networking.k8s.io/*`, core `Namespaces`, or
`cozystack.io/Package`. The protections below split into three groups by
who they defend against:
**Tenant-user-input gates.** Layer 4 (`cozystack-tenant-host-policy`).
`Tenant.spec.host` is the user-supplied field that surfaces as a
security boundary at the hostname layer; gated on every Create / Update
via `cozystack-api`'s admission chain.
**Defense-in-depth.** Layers 1, 2, 5, 7. These do not protect against
tenant-user input (tenants don't hold the relevant RBAC). They guard
against bugs in cozystack-controller / Flux, supply-chain compromise of
an app chart that emits Gateway API resources, and confused-deputy
mistakes by a cluster admin. Fail-closed via `failurePolicy: Fail` +
`validationActions: [Deny]`.
The five layers themselves:
1. **Listener `allowedRoutes` label selector** —
`namespace.cozystack.io/gateway = <owner-tenant-name>`
(controller-written for cozy-* via `AttachedNamespaces`, Helm-written
for tenant namespaces via the tenant chart, both feeding into the same
Gateway selector).
2. **`cozystack-gateway-hostname-policy`** — VAP on `Gateway`
CREATE/UPDATE. Restricts listener hostnames to the namespace's
`namespace.cozystack.io/host` apex.
4. **`cozystack-tenant-host-policy`** — VAP on `Tenant` CREATE/UPDATE.
5. **`cozystack-namespace-host-label-policy`** — VAP on core `Namespace`
CREATE/UPDATE. Immutability gate on the host label.
7. **`cozystack-route-hostname-policy`** — VAP on `HTTPRoute` and
`TLSRoute` CREATE/UPDATE. Closes the hostname-hijack vector across
namespaces under inheritance — a child cannot claim a hostname outside
its own apex.
The previous Layer 3 (`cozystack-gateway-attached-namespaces-policy`)
and Layer 6 (render-time `fail` in cozystack-basics) banned `tenant-*`
entries in `publishing.gateway.attachedNamespaces` on the assumption
that the attach list was the hijack vector. Under inheritance the attach
surface is the label selector, not the static list; the hijack vector is
closed by Layers 4/5/7 independently. Both Layer 3 and Layer 6 are
dropped — kept in numbering above for continuity with the earlier review
threads.
## Foreign-takeover guards
Five reconcile paths refuse to silently rewrite a pre-existing object
that shares the controller-derived name but carries no `OwnerReference`
back to the TenantGateway: `Gateway`, redirect `HTTPRoute`, per-tenant
`Issuer`, wildcard `Certificate`, per-listener `Certificate`. An
operator who hand-pinned a Certificate or Issuer at the controller's
derived name (private CA, manual cert pinning) gets an explicit
`Ready=False/ReconcileError` condition instead of having their config
silently destroyed and the resource re-issued from a different ACME
account.
The namespace-label patching path applies the same discipline:
cozystack-controller only writes / strips
`namespace.cozystack.io/gateway` on namespaces it annotates with
`cozystack.io/gateway-attached-by`. Labels written by the apps/tenant
chart (no annotation) are never touched, so inheritance for tenant
namespaces survives every reconcile.
## Tests
- **Go controller tests** — 50+ cases in
`internal/controller/tenantgateway/`: idempotency, OwnerReference
cascade, multi-parentRef status, HTTP-01↔DNS-01 mode-transition cert
lifecycle, foreign-takeover refusals (Gateway / HTTPRoute / Issuer /
Certificate / per-listener Certificate), DNS-01 provider matrix
(cloudflare / route53 / digitalocean / rfc2136), HTTPS Kinds=[HTTPRoute]
restriction, fail-closed CEL on missing host label, status condition
wiring. Plus, for inheritance: label-based allowedRoutes selector shape,
namespace label patch + garbage-collect via owner annotation, Helm-owned
label safety, DNS-01 wildcard SAN expansion per inheriting child,
per-child wildcard listener materialisation.
- **Admission-chain tests** —
`pkg/registry/apps/application/rest_admission_test.go` pins
`createValidation`, `updateValidation`, `deleteValidation` paths via
sentinel callbacks.
- **helm-unittest** across `extra/gateway` (TenantGateway CR rendering,
both modes), `core/platform` (DNS-01 wiring), `apps/tenant`
(gateway-effective decision matrix including inheritance),
`system/cozystack-basics` (three VAPs + Layer 7 VAP + tenant-root
namespace label including the conditional
`namespace.cozystack.io/gateway: tenant-root` + cozystack-values-secret
`_namespace.gateway` propagation), `system/cert-manager-issuers` (full
DNS-01 provider matrix), `system/cozystack-api` and `system/dashboard`
(Ingress↔TLSRoute/HTTPRoute toggle), `system/monitoring` (Grafana /
Alerta Ingress↔HTTPRoute toggle).
- **e2e bats** — `hack/e2e-apps/gateway.bats` (17 cases) covering
GatewayClass Accepted, Gateway Programmed + LB Service, HTTPRoute
Accepted with status, VAP rejects cross-tenant hostname hijack, VAP
rejects `tenant.spec.host` writes from non-trusted SA
(regression-detector for the admission-chain fix), VAP rejects
`namespace.cozystack.io/host` label changes from non-trusted SA,
route-hostname VAP rejects HTTPRoute hostnames outside the apex, and
three new cases for inheritance — Package admission accepts `tenant-*`
in `attachedNamespaces` (regression guard for the dropped ban), child
tenant without explicit gateway inherits `_namespace.gateway` from a
Gateway-owning parent in lockstep across the namespace label and the
cozystack-values Secret, and the cross-namespace attach end-to-end where
a child tenant's HTTPRoute reaches Accepted on the parent's Gateway via
the inheritance label.
- **CI wiring** — `make test-controllers` target invoked from
`pull-requests.yaml` runs `go test ./internal/...`.
## Backward compatibility
The only always-on change is the flip of `cilium.envoy.enabled` +
`cilium.gatewayAPI.enabled` to `true` (extra `cilium-envoy` DaemonSet,
~100 MB RAM/node idle). Everything else is gated behind
`gateway.enabled=false` / `tenant.spec.gateway=false`. Existing clusters
see no behavioural change until an operator explicitly opts in.
The cozystack-api admission-chain fix is unconditional. Existing
operators who do not define any VAP / admission webhook on
`apps.cozystack.io` resources see no observable change. Operators who DO
have their own webhooks/VAPs against these kinds will start seeing them
fire on Create and Delete (as intended) — review your policies before
merging if you maintain custom admission for `apps.cozystack.io/*`.
Operators who set `publishing.gateway.attachedNamespaces` to include
`tenant-*` entries used to hit a render-time fail or an admission
rejection — now the apply succeeds and the listed tenant namespaces
simply pick up the gateway-attach label alongside the cozy-* system
namespaces. This matches the inheritance model and was kvaps' explicit
ask in the May 25 review.
## Release note
```release-note
- Adds Gateway API support via Cilium, materialised per-tenant via the new `gateway.cozystack.io/v1alpha1 TenantGateway` CRD reconciled by `cozystack-controller`. Opt-in via `publishing.gateway.enabled=true` at the platform level; per-tenant opt-in via `tenant.spec.gateway=true` for dedicated Gateway / LB IP, otherwise the tenant inherits the nearest ancestor's Gateway via a label-based `allowedRoutes` selector (same shape as the existing `_namespace.ingress` inheritance). Includes HTTP-01 (default) and DNS-01 (opt-in, with cloudflare / route53 / digitalocean / rfc2136 providers) cert solvers; DNS-01 extends the parent's wildcard Certificate with per-child apex SANs and adds a `*.<child-apex>` listener per inheriting tenant. Tenant API is mechanism-agnostic for IP allocation — the LoadBalancer Service draws its IP from the cluster's admin-configured allocator (MetalLB / Cilium LB-IPAM / robotlb / externalIPs).
- Strengthens admission coverage on `apps.cozystack.io/*` resources: `cozystack-api` now invokes `createValidation` and `deleteValidation` callbacks on Create and Delete (Update was already wired). Operators with custom ValidatingAdmissionPolicies or webhooks targeting `apps.cozystack.io/*` will see them fire on all three verbs as the API contract requires; review your policies before merging if you maintain such admission.
```
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous commit (6498d37b) added `USER 65532:65532` to seven
first-party scratch images. Two of them — cozystack-api and
lineage-controller-webhook — mount their TLS serving certs as a
secret with `defaultMode: 0400` (root-only). With USER 65532, the
process can no longer read `tls.crt` / `tls.key` and the webhook
pod fails to start, manifesting downstream as
`failed calling webhook "lineage.cozystack.io": dial tcp <ServiceIP>:443: connect: operation not permitted`
during install of the tenant-root HelmRelease in e2e.
Reverts the USER directive on those two images. Properly switching them
to non-root requires either flipping the secret mount to
`defaultMode: 0440` plus `securityContext.fsGroup: 65532` on the
pod, or relaxing to 0444 — both choices touch the runtime chart and
are out of scope for a Go-deps security PR. Tracked as a follow-up.
The remaining five hardened images (backup-controller,
backupstrategy-controller, token-proxy, flux-plunger, kubeovn-plunger)
do not mount root-only secrets and continue to run as 65532.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
CodeRabbit (PR review on #2494) flagged that the first-party scratch
images leave the container process running as root because no USER
directive is set. Adds the standard distroless nonroot UID/GID
65532:65532 just before ENTRYPOINT in seven first-party Dockerfiles.
The chosen UID matches gcr.io/distroless/static:nonroot and the
existing convention in the kubevirt-csi-driver and kamaji images.
Statically-linked Go binaries copied with default permissions remain
world-readable, so they execute fine under the non-root identity
without an extra chown step in the builder stage.
Touched:
- packages/system/backup-controller/images/backup-controller/Dockerfile
- packages/system/backupstrategy-controller/images/backupstrategy-controller/Dockerfile
- packages/system/cozystack-api/images/cozystack-api/Dockerfile
- packages/system/dashboard/images/token-proxy/Dockerfile
- packages/system/flux-plunger/images/flux-plunger/Dockerfile
- packages/system/kubeovn-plunger/images/kubeovn-plunger/Dockerfile
- packages/system/lineage-controller-webhook/images/lineage-controller-webhook/Dockerfile
Vendored Dockerfiles (kubevirt-cloud-provider, redis-operator,
keycloak-operator, linstor-csi, multus-cni, objectstorage,
s3manager, kamaji, cluster-autoscaler) are intentionally left
untouched — their runtime user choice is upstream's call.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The root go.mod now declares go 1.26.2 (raised in the first commit
of this PR). Every Dockerfile that copies the root go.mod and builds
a cozystack-owned binary has to use a matching base image, otherwise
the in-container go toolchain has to download go 1.26 on the fly
(via GOTOOLCHAIN=auto) at every build, which slows builds and breaks
in network-restricted CI sandboxes.
Bumps:
- packages/core/installer/images/cozystack-operator
- packages/system/backup-controller/images/backup-controller
- packages/system/backupstrategy-controller/images/backupstrategy-controller
- packages/system/cozystack-api/images/cozystack-api
- packages/system/cozystack-controller/images/cozystack-controller
- packages/system/flux-plunger/images/flux-plunger
- packages/system/kubeovn-plunger/images/kubeovn-plunger
- packages/system/lineage-controller-webhook/images/lineage-controller-webhook
All eight images go from golang:1.25-alpine to golang:1.26-alpine.
Also normalizes the cozystack-operator Dockerfile FROM stage keyword
from lowercase 'as' to uppercase 'AS' while touching it.
Vendored Dockerfiles (kubevirt-cloud-provider, redis-operator,
keycloak-operator, linstor-csi, multus-cni, objectstorage,
s3manager, kamaji) are intentionally untouched: they pull upstream
sources at build time and pin their go base image to whatever the
upstream commit was tested against.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace DaemonSet with direct host API access in favor of a regular
Deployment using Service trafficDistribution: PreferClose. This provides
prefer-local routing to the nearest cozystack-api pod with fallback
to remote pods when local one is unavailable.
- Replace DaemonSet/Deployment toggle with always-Deployment
- Replace internalTrafficPolicy: Local with trafficDistribution: PreferClose
- Remove KUBERNETES_SERVICE_HOST/PORT override (use default kubernetes service)
- Replace hard nodeSelector with soft nodeAffinity for control-plane nodes
- Simplify migration hook to always clean up DaemonSet if present
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Same issue as lineage-controller-webhook: DaemonSet uses hardcoded
nodeSelector with empty value, but k3s/kubeadm use value "true".
Changes:
- Add nodeSelector to cozystack-api values (default: empty for Talos)
- Update deployment template to use configurable nodeSelector
- Pass nodeSelector for both cozystack-api and lineage-controller-webhook
in isp-full-generic bundle variant
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Cozystack-api and lineage-webhook tolerate all taints, so they are able
to run on masters no matter what.
Needed for unschedulable control-plane setup, quorum nodes, etc.
### Release note
```release-note
Cozystack-api and lineage-webhook tolerate all taints.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated pod scheduling tolerations to improve deployment flexibility
and compatibility across diverse cluster configurations.
* Enhanced workload distribution by making pods more resilient to
cluster taints, enabling better resource utilization in various
environments.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch fixes an issue with the Cozystack API server that causes it
to respond with the first registered dynamic list kind when listing
dynamic resources. E.g., when running `k get postgreses`, the raw JSON
response from the cozystack API would be
```json
{
"apiVersion":"apps.cozystack.io/v1alpha1",
"kind":"BootBoxList",
"items": [
{
"apiVersion":"apps.cozystack.io/v1alpha1",
"kind":"Postgres",
...
},
...
],
...
}
```
The root cause is the way the `Typer` interface is implemented for the
`runtime.Scheme` where the dynamic types are registered. Since the base
type for all dynamic types is a `&cozyv1alpha1.Application{}`, all types
are registered under the same key and the `Typer` defaults to the first
`GroupVersionKind` that was registered. Only when a correctly formed
`&unstructured.Unstructured{}` is returned by the API, is this resolving
logic circumvented and the `GroupVersionKind` is instead inferred from
the fields of the returned object. Even an `UnstructuredList` is not
acceptable as a return type, instead the `items` key should be directly
set on the underlying `map[string]interface{}`.
This patch implements the changes detailed above. Additionally, the
following features, fixes, and improvements are added:
* Makefile targets to build and run the Cozystack API locally, against a
Kubernetes server in the environment's KUBECONFIG. Debugging with
Delve is also supported.
* CI tests are added to verify the new changes.
* A bug in the registration of the corev1alpha1 types is fixed.
* Updated the `ConvertToTable` function to properly handle list kinds
which might be of the `&unstructured.Unstructured{}` concrete type
(not an `UnstructuredList`).
* The scheme used by the API server's Client and WatchClient is
separated from the scheme used to serve dynamic types.
* The client config for reading the runtime configuration now uses the
controller-runtime, which handles flags and environment variables
properly, unlike `clientcmd`.
### Release note
```release-note
[api] Fix incorrect list kind for list requests to the Cozystack API for
dynamic resources. Add Makefile targets for local testing. Minor schema
building improvements.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch updates the default API Ingress to add the
nginx.ingress.kubernetes.io/force-ssl-redirect annotation,
ensuring all HTTP traffic (port 80) is redirected to HTTPS (port 443).
This prevents unencrypted external access and improves security.
```release-note
[ingress] Force HTTPS access for api.dev3.infra.aenix.org and block direct HTTP.
```
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
It was observed during upgrades to the `cozystack-api` Helm release that
when enabling the local endpoint for the traffic locality feature, hence
switching from a deployment to a daemonset, the deployment may remain
unpruned and the pods of the deployment will continue to run
indefinitely. This patch adds a post-upgrade hook that explicitly deletes
the deployment in case it exists and was not pruned.
### Release-note
```release-note
[api] Delete the cozystack-api deployment in a post-upgrade hook when
migrating to a daemonset and vice-versa.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Since 0.37, many requests to the k8s API now go through a mutating
webhook (lineage-controller-webhook). Since the lineage webhook makes
multiple requests to the k8s API and, indirectly, to the Cozystack API
server, each request for, e.g., creating a secret now causes a lot of
chatter between the webhook, the k8s API, and the Cozystack API. When
this happens cross-node or, worse yet, cross-zone, this can blow up the
latency for simple requests.
This patch changes the Cozystack API to a DaemonSet targetting
controlplane nodes, configures its service for an `Local` internal
traffic policy and adds environment variables indicating that the k8s
API server is to be found at <hostIP>:6443, **not only for the Cozystack
API, but also for the lineage-controller-webhook.** This is a valid
configuration in most scenarios, including the default installation
method on top of Talos Linux in Cozystack, however, if this is not valid
in your environment, you must now set the values
`.lineageControllerWebhook.localK8sAPIEndpoint.enabled` and
`.cozystackAPI.localK8sAPIEndpoint.enabled` to `false` in the respective
system Helm releases.
```release-note
[api,lineage] Configure all chatter between the Lineage webhook, the
Cozystack API server and the Kubernetes API server to be confined to a
single controlplane node, improving k8s API latency.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
cozystack-ci[bot]