Every release-1.5 PR Build job dies at the 30-minute timeout, and all of
it is spent in one target. `image-ubuntu-container-disk` builds a disk
per Kubernetes minor -- six of them -- and each spends ~4-5min inside a
libguestfs appliance installing that minor's kubelet/kubeadm into its own
copy of the cloud image. Serially that is ~28min of the 30min budget:
v1.35 6:00, v1.34 4:21, v1.33 4:27, v1.32 4:39, v1.31 4:42, v1.30 cancelled
None of it is cached, and no warmer will fix that on its own. Only
build-main.yaml sets WRITE_CACHE=1, and main dropped this image from
`make image` in 39cdc8866 six days before release-1.5 was cut, so the
:*-buildcache refs it reads have never been written -- the CI log shows
all six importers failing with "not found". release-1.5 is now the only
branch that builds this image at all.
The six builds share no per-version work, so run them concurrently
instead. A sub-make carries its own -j, keeping the root `make build`
serial, and buildkit dedups the shared guestfish and cloud-image stages
across the concurrent solves. --output-sync=target groups each version's
log under its target name; six interleaved buildx progress streams have
no per-solve identifier and are unreadable. Verified the per-version
buildx/echo/rm command set is byte-identical to the loop it replaces
across PUBLISH_VERSIONED, PUBLISH_FLOATING and WRITE_CACHE combinations.
Moving off the 4cpu/16gb shape because six concurrent libguestfs
appliances want ~9gb of RAM and a core apiece. The comment justifying
the small shape assumed a warm cache, which no longer holds here.
This also stops swallowing failures. The loop separated versions with
`;`, so make saw only the last version's exit status: a failed build for
any other version left its committed .tag file stale and the job green.
Under the sub-make a failure in any version fails the target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds **test-impact analysis (TIA)** to the PR E2E workflow so that only
bats files affected by the diff are exercised, and a **release-time E2E
workflow** that runs the full suite on every release tag to keep
coverage on the shipped code.
### 1. Test-impact analysis selector — `hack/select-e2e.sh`
Reads `packages/core/platform/sources/*.yaml` (the PackageSource
dependency graph) and emits the bats files affected by a PR diff. The
walk is the same one `cozypkg dot` renders.
| Input | Output |
|---|---|
| `packages/apps/postgres/values.yaml` | `postgres` |
| `packages/system/postgres-operator/...` | `postgres harbor`
(transitive) |
| `packages/system/cilium/...` | full suite (cilium has many transitive
`*-application` dependents) |
| `packages/library/cozy-lib/...` | full suite (library affects
everything) |
| `hack/e2e-install-cozystack.bats` | full suite (shared install affects
all apps) |
| `hack/e2e-apps/redis.bats` | `redis` (per-app edit, never escalates) |
| `docs/agents/contributing.md` | (empty — no E2E impact) |
| `dashboards/gpu/gpu-fleet.json` | (empty) |
Conservative fallbacks:
- An unrecognised `packages/*` path or a system source with no
`*-application` descendants escalates to the full suite, so a path
inside the graph is never silently dropped.
- Per-app bats edits are matched **before** the full-suite trigger, so
editing one bats file selects only that app rather than escalating.
### 2. Default-on TIA + `full-e2e` opt-out — `pull-requests.yaml`
- The `Select E2E tests` step runs by default and produces `apps` +
`skip` outputs.
- `Run E2E tests` runs unless `skip=true` (docs / dashboards / `*.md`
only).
- Adding the `full-e2e` label skips the selector step (output empty) and
falls through to the unchanged full-bats-list path — the safety hatch
when reviewers want the full suite on a PR.
### 3. `release-e2e.yaml` — full E2E on every release tag
Triggered on `v*.*.*`, `v*.*.*-rc.*`, `v*.*.*-beta.*`, `v*.*.*-alpha.*`.
Builds images + Talos image, then runs the full suite. Independent of
`tags.yaml`'s release publish flow — failure here is an alarm, not a
release blocker.
This closes the coverage gap left by making PR runs default to TIA: the
full suite is now exercised at tag-cut time against the shipped code.
## Tests
`hack/select-e2e_test.bats` covers 11 cases — single app, transitive
operator, networking/library full-suite, docs/dashboards skip,
`kubernetes-application` two-bats mapping, shared E2E helper, install
bats, per-app bats no-escalation, release-e2e workflow change.
YAML parse verified for both new workflow files.
## Workflow injection safety
All `${{ }}` contexts that could carry untrusted data
(`github.base_ref`, label arrays) flow through env vars, never directly
into shell text. `github.workspace` is also indirected through
`WORKSPACE` env in the release workflow.
### Release note
```
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* CI now runs a Test Impact Analysis to pick only affected apps,
reducing E2E runtime.
* Added a release-level E2E workflow that runs full acceptance tests for
version tags with deterministic sandboxing, artifact capture,
conditional retries, teardown, and report upload.
* E2E jobs accept selected-apps metadata and support conditional
full-suite overrides.
* **Tests**
* New test suite validates selection logic, operator expansion,
full-suite escalation, and docs-only/no-op behaviors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Run 27057500380 failed on harbor's BucketClaim never reaching
bucketReady=true, and neither artifact could explain why: the per-app
diagnostics dump has no bucket objects, and cozyreport collects
neither COSI resources nor objectstorage-controller / seaweedfs
provisioner logs — by the time the report is taken, the bucket state
has moved on.
Add a COSI module to cozyreport.sh (bucketclaims/bucketaccesses/
buckets/classes as custom-column tables plus full YAML, controller and
per-namespace provisioner logs) and a bucket-readiness snapshot to the
per-app failure dump in the PR workflow. The COSI CRDs ship no printer
columns, so plain kubectl get would show only NAME/AGE — readiness
fields are pulled explicitly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Mirror the diagnostics group already on `Run E2E tests` to the
`Install Cozystack into sandbox` step. Without this, install-step
failures (e.g. an HR not reaching Ready inside the bats `kubectl wait`
window) surface only as "error: timed out waiting for the condition
on helmreleases/<name>" with no cluster snapshot — re-running is the
only way to triage, which is the exact pattern the parent commit
removed.
On failure, dump three groups:
- `kubectl get hr -A -o wide` (last 100 lines): which HRs exist and
their Ready/Status columns.
- `kubectl describe hr` for each not-Ready HR (tail -50): the
Conditions block surfaces helm install/upgrade errors, missing
CRDs, and chart-render failures — the actual cause.
- `kubectl get events -A --sort-by=.lastTimestamp` (last 50): cluster
events that often pinpoint the failing pod/PVC/job.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from coderabbitai on
.github/workflows/pull-requests.yaml:235: $attempt is incremented only on
failure, so first-try success printed "completed after 0 attempts". Use
$((attempt + 1)) so the message reflects the total number of runs.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry
loop never recovered a flake, only stretched deterministic failures
and tripled diagnostic wall-time. Same data shape on `Install
Cozystack`. Drop both.
`Prepare environment` keeps its 3x retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network)
where transient runner hiccups warrant a retry.
On failure, the test step now captures `kubectl get hr -A -o wide`
and `kubectl get events -A` under a collapsible group so triage
starts with the actual broken-state snapshot.
Depends on:
- #2508 — installer namespace bootstrap (Helm namespace-ownership conflict)
- #2509 — operator HelmRelease config knobs (`seaweedfs-system` 2-min wait race)
- #2528 — harbor bucket-secret + BucketInfo gating (harbor ValuesError)
- #2529 — objectstorage-controller BucketAccess conflict retry
- the daniil/split-vminstance PR (vminstance disk race + VM IP/ready timeouts)
- the daniil/split-event-driven PR (existence backstops surfacing real errors)
Until those land, dropping the retry will fail CI for unrelated PRs
that hit the seaweedfs / harbor / installer / vminstance races.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since 23e399bd, but `Create` and `Delete` accepted the
parameter and never called it — admission was silently bypassed for both
verbs on every `apps.cozystack.io` resource (Tenant, MariaDB, Postgres,
Kubernetes, …).
This PR wires the missing `createValidation` and `deleteValidation`
calls into Create and Delete. Every ValidatingAdmissionPolicy and
ValidatingWebhook against `apps.cozystack.io/*` now fires on all three
verbs as the API contract requires. Per KEP-3488 aggregated APIServers
are responsible for enforcing admission on their own kinds;
cozystack-api is configured to do so via
`genericoptions.NewRecommendedOptions`, the only thing missing was the
explicit `*Validation` callbacks in the custom REST.
## Security — five guards, grouped by what they defend
Tenants in Cozystack interact with the platform exclusively through
`apps.cozystack.io/*` resources (Tenant, Bucket, Kubernetes, …) served
by `cozystack-api`. Tenant RBAC (`cozy:tenant:*` aggregated to a
RoleBinding in the tenant's own namespace) does not grant write access
to `gateway.networking.k8s.io/*`, core `Namespaces`, or
`cozystack.io/Package`. The protections below split into three groups by
who they defend against:
**Tenant-user-input gates.** Layer 4 (`cozystack-tenant-host-policy`).
`Tenant.spec.host` is the user-supplied field that surfaces as a
security boundary at the hostname layer; gated on every Create / Update
via `cozystack-api`'s admission chain.
**Defense-in-depth.** Layers 1, 2, 5, 7. These do not protect against
tenant-user input (tenants don't hold the relevant RBAC). They guard
against bugs in cozystack-controller / Flux, supply-chain compromise of
an app chart that emits Gateway API resources, and confused-deputy
mistakes by a cluster admin. Fail-closed via `failurePolicy: Fail` +
`validationActions: [Deny]`.
The five layers themselves:
1. **Listener `allowedRoutes` label selector** —
`namespace.cozystack.io/gateway = <owner-tenant-name>`
(controller-written for cozy-* via `AttachedNamespaces`, Helm-written
for tenant namespaces via the tenant chart, both feeding into the same
Gateway selector).
2. **`cozystack-gateway-hostname-policy`** — VAP on `Gateway`
CREATE/UPDATE. Restricts listener hostnames to the namespace's
`namespace.cozystack.io/host` apex.
4. **`cozystack-tenant-host-policy`** — VAP on `Tenant` CREATE/UPDATE.
5. **`cozystack-namespace-host-label-policy`** — VAP on core `Namespace`
CREATE/UPDATE. Immutability gate on the host label.
7. **`cozystack-route-hostname-policy`** — VAP on `HTTPRoute` and
`TLSRoute` CREATE/UPDATE. Closes the hostname-hijack vector across
namespaces under inheritance — a child cannot claim a hostname outside
its own apex.
The previous Layer 3 (`cozystack-gateway-attached-namespaces-policy`)
and Layer 6 (render-time `fail` in cozystack-basics) banned `tenant-*`
entries in `publishing.gateway.attachedNamespaces` on the assumption
that the attach list was the hijack vector. Under inheritance the attach
surface is the label selector, not the static list; the hijack vector is
closed by Layers 4/5/7 independently. Both Layer 3 and Layer 6 are
dropped — kept in numbering above for continuity with the earlier review
threads.
## Foreign-takeover guards
Five reconcile paths refuse to silently rewrite a pre-existing object
that shares the controller-derived name but carries no `OwnerReference`
back to the TenantGateway: `Gateway`, redirect `HTTPRoute`, per-tenant
`Issuer`, wildcard `Certificate`, per-listener `Certificate`. An
operator who hand-pinned a Certificate or Issuer at the controller's
derived name (private CA, manual cert pinning) gets an explicit
`Ready=False/ReconcileError` condition instead of having their config
silently destroyed and the resource re-issued from a different ACME
account.
The namespace-label patching path applies the same discipline:
cozystack-controller only writes / strips
`namespace.cozystack.io/gateway` on namespaces it annotates with
`cozystack.io/gateway-attached-by`. Labels written by the apps/tenant
chart (no annotation) are never touched, so inheritance for tenant
namespaces survives every reconcile.
## Tests
- **Go controller tests** — 50+ cases in
`internal/controller/tenantgateway/`: idempotency, OwnerReference
cascade, multi-parentRef status, HTTP-01↔DNS-01 mode-transition cert
lifecycle, foreign-takeover refusals (Gateway / HTTPRoute / Issuer /
Certificate / per-listener Certificate), DNS-01 provider matrix
(cloudflare / route53 / digitalocean / rfc2136), HTTPS Kinds=[HTTPRoute]
restriction, fail-closed CEL on missing host label, status condition
wiring. Plus, for inheritance: label-based allowedRoutes selector shape,
namespace label patch + garbage-collect via owner annotation, Helm-owned
label safety, DNS-01 wildcard SAN expansion per inheriting child,
per-child wildcard listener materialisation.
- **Admission-chain tests** —
`pkg/registry/apps/application/rest_admission_test.go` pins
`createValidation`, `updateValidation`, `deleteValidation` paths via
sentinel callbacks.
- **helm-unittest** across `extra/gateway` (TenantGateway CR rendering,
both modes), `core/platform` (DNS-01 wiring), `apps/tenant`
(gateway-effective decision matrix including inheritance),
`system/cozystack-basics` (three VAPs + Layer 7 VAP + tenant-root
namespace label including the conditional
`namespace.cozystack.io/gateway: tenant-root` + cozystack-values-secret
`_namespace.gateway` propagation), `system/cert-manager-issuers` (full
DNS-01 provider matrix), `system/cozystack-api` and `system/dashboard`
(Ingress↔TLSRoute/HTTPRoute toggle), `system/monitoring` (Grafana /
Alerta Ingress↔HTTPRoute toggle).
- **e2e bats** — `hack/e2e-apps/gateway.bats` (17 cases) covering
GatewayClass Accepted, Gateway Programmed + LB Service, HTTPRoute
Accepted with status, VAP rejects cross-tenant hostname hijack, VAP
rejects `tenant.spec.host` writes from non-trusted SA
(regression-detector for the admission-chain fix), VAP rejects
`namespace.cozystack.io/host` label changes from non-trusted SA,
route-hostname VAP rejects HTTPRoute hostnames outside the apex, and
three new cases for inheritance — Package admission accepts `tenant-*`
in `attachedNamespaces` (regression guard for the dropped ban), child
tenant without explicit gateway inherits `_namespace.gateway` from a
Gateway-owning parent in lockstep across the namespace label and the
cozystack-values Secret, and the cross-namespace attach end-to-end where
a child tenant's HTTPRoute reaches Accepted on the parent's Gateway via
the inheritance label.
- **CI wiring** — `make test-controllers` target invoked from
`pull-requests.yaml` runs `go test ./internal/...`.
## Backward compatibility
The only always-on change is the flip of `cilium.envoy.enabled` +
`cilium.gatewayAPI.enabled` to `true` (extra `cilium-envoy` DaemonSet,
~100 MB RAM/node idle). Everything else is gated behind
`gateway.enabled=false` / `tenant.spec.gateway=false`. Existing clusters
see no behavioural change until an operator explicitly opts in.
The cozystack-api admission-chain fix is unconditional. Existing
operators who do not define any VAP / admission webhook on
`apps.cozystack.io` resources see no observable change. Operators who DO
have their own webhooks/VAPs against these kinds will start seeing them
fire on Create and Delete (as intended) — review your policies before
merging if you maintain custom admission for `apps.cozystack.io/*`.
Operators who set `publishing.gateway.attachedNamespaces` to include
`tenant-*` entries used to hit a render-time fail or an admission
rejection — now the apply succeeds and the listed tenant namespaces
simply pick up the gateway-attach label alongside the cozy-* system
namespaces. This matches the inheritance model and was kvaps' explicit
ask in the May 25 review.
## Release note
```release-note
- Adds Gateway API support via Cilium, materialised per-tenant via the new `gateway.cozystack.io/v1alpha1 TenantGateway` CRD reconciled by `cozystack-controller`. Opt-in via `publishing.gateway.enabled=true` at the platform level; per-tenant opt-in via `tenant.spec.gateway=true` for dedicated Gateway / LB IP, otherwise the tenant inherits the nearest ancestor's Gateway via a label-based `allowedRoutes` selector (same shape as the existing `_namespace.ingress` inheritance). Includes HTTP-01 (default) and DNS-01 (opt-in, with cloudflare / route53 / digitalocean / rfc2136 providers) cert solvers; DNS-01 extends the parent's wildcard Certificate with per-child apex SANs and adds a `*.<child-apex>` listener per inheriting tenant. Tenant API is mechanism-agnostic for IP allocation — the LoadBalancer Service draws its IP from the cluster's admin-configured allocator (MetalLB / Cilium LB-IPAM / robotlb / externalIPs).
- Strengthens admission coverage on `apps.cozystack.io/*` resources: `cozystack-api` now invokes `createValidation` and `deleteValidation` callbacks on Create and Delete (Update was already wired). Operators with custom ValidatingAdmissionPolicies or webhooks targeting `apps.cozystack.io/*` will see them fire on all three verbs as the API contract requires; review your policies before merging if you maintain such admission.
```
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace MAINTAINERS.md snapshot with a list derived from actual recent
activity on main (commits, merged PRs and reviews over the last three
months). MAINTAINERS.md had drifted in both directions — listing people
no longer active on the project while missing several current
maintainers — so copying it verbatim was both handing SSH access to
inactive accounts and locking out people who would actually use this
debug feature.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
When the e2e job fails on a PR carrying the `debug` or `release` label
(release PRs are bot-created, so the label implies maintainer authorship),
hold the runner and open an SSH breakpoint via cozystack/breakpoint-action
(a fork of namespacelabs/breakpoint-action pinned by SHA) so a maintainer
can attach, inspect Talos/Cozystack state, and resume.
The fork adds three things on top of upstream:
* `mode: pause-idle` — waits 20m for the first SSH connection, then keeps
the workflow paused while sessions are active, exiting 10m after the
last session disconnects. Saves runner-minutes vs a fixed timeout.
* `endpoint` step output + `::notice::` annotation — surfaces the SSH
endpoint while the step is still paused (step logs are buffered until
step completion, so the endpoint would otherwise be invisible).
* A dedicated Check Run "Breakpoint Open" with conclusion=failure
carrying the SSH endpoint and sandbox-entry hint in output.summary
(Markdown, rendered in the UI). Surfaces as a standard ✗ in
`gh pr checks` so any CI watcher sees the open breakpoint without
bespoke marker parsing. Updated to conclusion=success when the
breakpoint exits.
The step is configured via the `BREAKPOINT_ENDPOINT` repo variable. In
fork PRs, repo variables are not exposed, so the step is naturally
skipped and forks cannot reach the self-hosted rendezvous server. The
`authorized-users` list is kept as a second defense layer at the
breakpoint level.
Permissions on the e2e job: `checks: write` (for Check Run create+update).
Upstream PR: namespacelabs/breakpoint-action#16
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The 2-hour ceiling on the E2E job was clipping otherwise-working
runs on the oracle-vm-24cpu-96gb-x86-64 runners: image pulls,
multi-stage Talos bootstrap, full helm-release fan-out plus the
post-install bats matrix routinely consume 100-180 minutes of
wall-clock, and runs that would have eventually gone green were
getting cancelled by GitHub at the timeout.
Recent successful E2E runs on the same runner pool sit between
60 and 184 minutes; lift the cap to 180m to cover the long tail
without inviting truly-stuck jobs to linger forever.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The "Select E2E tests" step runs `git diff --name-only
origin/${BASE_REF}...HEAD` inside the sandbox copy of the workspace.
Without fetch-depth: 0 the e2e job's checkout only resolves a single
commit, so origin/main is missing and the diff aborts with
"ambiguous argument 'origin/main...HEAD': unknown revision".
Mirror the Build job's setup and request full history.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes that compose into a single E2E policy:
1. **Test-impact analysis selector** (`hack/select-e2e.sh`).
Reads `packages/core/platform/sources/*.yaml` (the PackageSource
dependency graph also rendered by `cozypkg dot`) and emits the bats
files affected by a PR diff. Library/core/networking/CI changes,
shared `hack/*.sh` helpers, the install bats, and the
`pull-requests` / `release-e2e` workflows trigger the full suite.
Doc/dashboard/`*.md` diffs select nothing. Per-app bats edits
select only that app. Anything inside `packages/` with no graph
entry, or a system source whose graph has no `*-application`
descendants, falls back to the full suite (conservative).
2. **Default-on TIA in `pull-requests.yaml`** with `full-e2e`
opt-out. Drops the previous `e2e-tia` opt-in shape — the
selector's behaviour has been validated, so making it the default
gives PR runtime savings out of the box. PRs that need the full
suite (e.g. cross-cutting refactors that the selector can't
reason about) carry the `full-e2e` label and get the unchanged
pre-TIA flow. Docs-only PRs (selector emits nothing) skip the
`Run E2E tests` step entirely.
3. **`release-e2e.yaml`** runs the full suite on every tag push
(`v*`). Independent of `tags.yaml`'s release publish flow — a
failure here is an alarm, not a release blocker. Closes the
coverage gap left by making PR runs use TIA: the full suite is
now exercised against shipped code at tag-cut time.
Bug fixes vs. the original `feat/select-e2e-script` draft:
- `dashboard/` -> `dashboards/` (path is plural in this repo).
- `hack/e2e-apps/<name>.sh` and `hack/<name>.bats` were silently
ignored; they now trigger the full suite (test infrastructure
affecting all apps).
- A path matching the graph but yielding zero `*-application`
descendants now falls back to the full suite instead of silently
emitting nothing (system-only changes are no longer silently
skipped).
- Per-app bats edits are matched before the full-suite pattern, so
editing one bats file no longer escalates to the full suite.
- Removed the dead `skip_pattern` variable (PCRE syntax that
`grep -qE` doesn't support; it was never matched).
Tests: `hack/select-e2e_test.bats` covers all 11 cases (the original
6 plus 5 for the bug fixes above).
Workflow inputs are passed via env vars rather than direct `${{ }}`
expansion into shell commands, per workflow injection guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
The GH_PAT secret tied to cozystack-bot is no longer valid after
migrating release workflows to the cozystack-ci GitHub App in #2351.
This broke the resolve_assets and e2e jobs for release PRs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Without explicit timeouts, stuck jobs use the GitHub Actions
default of 360 minutes (6 hours). This blocks the concurrency
group and prevents new runs from starting.
Set reasonable limits:
- Build: 30 minutes (normally ~5 min)
- E2E: 120 minutes (normally ~60 min)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pull-requests workflow used paths-ignore at the trigger level, which
prevented the entire workflow from running on docs-only PRs. This meant
the required "E2E Tests" check was never created, blocking merge for
non-admin users.
Replace trigger-level paths-ignore with a detect-changes job using
dorny/paths-filter. The workflow now always triggers (so checks are
always reported), but build and downstream jobs are skipped when only
docs files change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add -talos suffix to the default variant output file for consistency
with -generic and -hosted variants. Update all references in CI
workflows, e2e tests, upload scripts, and testing Makefile.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch adds a conditional for running on a statically defined VM the
maintainers have SSH access to if the pull request has a `debug` label.
This is useful for debugging failing workflows when the diagnostic info
from the pipeline is insufficient.
### Release note
```release-note
[ci] Run builds on a static VM with SSH access if the PR has a debug
label.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
- Add pattern rule for building cozypkg binaries per platform
(assets-cozypkg-<os>-<arch>) with checksums generation
- Add Version variable to cozypkg CLI injected via ldflags
- Split manifests into separate cozystack-crds.yaml and
cozystack-operator.yaml files
- Update CI workflow to handle CRDs and operator as separate artifacts
- Update e2e tests and upload script for new manifest structure
- Suppress git describe stderr when no tags exist
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
The cozy-lib library package got complicated enough to warrant its own
unit tests. Since unit tests are a "good thing" (tm), a somewhat generic
framework for running all kinds of unit tests was introduced into the CI
pipeline and Makefile targets. For now all it runs is `make test`
against the `packages/{library,apps,system,extra}/*` directories,
wherever a `test` target is present in the Makefile, and for now this is
only for the `cozy-lib` Helm library chart.
### Release note
```release-note
[ci,dx] Introduce a scaffold for running unit tests locally and in CI
and add the first unit tests for the cozy-lib helper Helm chart.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch sources the REGISTRY env var from GitHub actions variables
instead of secrets, so pull requests from forked repos work correctly.
```release-note
[ci] Source the REGISTRY env var from actions' variables, not secrets,
so external pull requests can work.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
* Simplify test discovery logic in workflow.
* Delete Clickhouse after successful test.
* Separate two k8s tests into separate jobs.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Changing the container registry from GHCR to OCIR will help with more
flexibe image retention policies and removes the restrictions on the
GitHub token when contributors submit PRs from forks. Release PRs remain
on GHCR, as before.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
- Skip long workflows on PRs that only change files inside the `./docs`
directory.
- Not applicable to other docs in this repository, such as
`packages/apps/**/*.md`, as they're part of the build.
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[ci] Skip Cozystack tests on PRs that only change the docs
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated automated workflows to skip running on pull requests that only
modify documentation files, reducing unnecessary workflow runs.
* Refined workflow triggers to exclude events triggered by labeling pull
requests, streamlining automation processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved reliability of automated testing workflows by adding retry
logic to key setup and test steps.
* Simplified resource management in end-to-end tests by switching to a
consistent apply command for creating or updating Kubernetes resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
I labeled my PR and CI was re-started, so now I have to wait even more.
We have no labels governing CI, so there's no reason to restart it on `labeled`.
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
- Skip long workflows on PRs that only change files inside `./docs` directory.
- Not applicable to other docs in this repository, such as `packages/apps/**/*.md`,
as they're part of the build.
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
This patch separates the Test job of the PR workflow into several
smaller jobs: 1) create a testing sandbox and deploy Talos, 2) install
Cozystack and configure it, 3) install managed applications and run e2e
tests. This lets developers shorten the feedback loop if tests are
merely acting flaky and aren't really broken. It's not the right way,
but it's 80/20.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Myasnikov Daniil