## What this PR does
Replaces ingress-nginx as the default ingress layer for cozystack-native
services with **Cilium Gateway API**, materialised per-tenant via a new
`TenantGateway` CRD reconciled by `cozystack-controller`. Supersedes
#2213 / #2208. Stacked on top of #2468.
## Architecture
The chart renders one `gateway.cozystack.io/v1alpha1 TenantGateway` CR
per tenant that owns its own Gateway. The cozystack-controller
reconciles the actual `Gateway`, per-tenant `Issuer`, `Certificate`
(wildcard or per-listener), and HTTP→HTTPS redirect `HTTPRoute` from
there. Helm does not render `Gateway` or `Certificate` directly — that
prevents the Helm-vs-controller race on `Gateway.spec.listeners` that
route-driven listener additions would otherwise cause.
```
┌── TenantGateway CR (rendered by extra/gateway chart for owning tenants)
│
└──▶ cozystack-controller reconciles:
├─ Gateway (with dynamic listeners, label-based allowedRoutes)
├─ Issuer (per-tenant ACME account)
├─ Certificate(s) — wildcard + per-child SANs in DNS-01, per-listener in HTTP-01
├─ HTTPRoute (controller-owned http→https redirect)
├─ Namespace label patching (cozy-* system namespaces from spec.AttachedNamespaces)
└─ Watches HTTPRoute / TLSRoute attachments to materialise listeners
```
## Inheritance: tenants attach by default, opt into their own Gateway
A tenant gets a dedicated Gateway (own LB Service, own LB IP, own
Certificate) only when it explicitly asks via
`tenant.spec.gateway=true`. Every other tenant in the tree publishes
through the Gateway of the nearest ancestor that owns one — same shape
as the existing `_namespace.ingress` inheritance.
The `apps/tenant` chart writes a `namespace.cozystack.io/gateway` label
on each tenant namespace, carrying either the tenant's own name (when
owning a Gateway) or the inherited ancestor name (when inheriting). The
Gateway's listener `allowedRoutes.namespaces.selector` is keyed on that
label, so the same selector admits routes from every namespace pointing
at the owner — descendants and cozy-* system namespaces alike.
cozystack-controller patches the same label onto every namespace in
`tgw.Spec.AttachedNamespaces` and garbage-collects labels it wrote when
an entry is removed from the attach list (annotation-gated GC so labels
written by the apps/tenant chart are never stripped).
In DNS-01 mode, the controller extends the parent's wildcard Certificate
with `<child-apex>` + `*.<child-apex>` SANs per inheriting child, and
adds a `*.<child-apex>` HTTPS listener per child apex referencing the
same cert. Without this expansion the parent's single-level wildcard
cannot match a child route's hostname (`harbor.alice.example.org` is two
labels past the parent's `*.example.org`).
Opt-out into own Gateway is the right choice when: a tenant needs its
own LB IP (DNS pinned, firewall rule), the apex is not derived from the
parent (custom `host` value the ancestor's cert can't cover), or the
tenant wants its own ACME account / cert authority. Otherwise leave
`gateway` unset and inherit.
## End-to-end flow
1. Platform: `gateway.enabled=true` toggles cert-manager solver,
switches per-system-app templates from Ingress to HTTPRoute / TLSRoute.
2. Per tenant: explicit `tenant.spec.gateway=true` renders a
`TenantGateway` CR and triggers the controller to materialise the owner
Gateway. Tenants with the field unset inherit through
`_namespace.gateway` propagation in the tenant chart.
3. Adding a published app under any tenant (owner or inheriting) is just
deploying its HTTPRoute. The controller picks up the hostname, adds a
per-listener Certificate (HTTP-01) or extends the wildcard cert's SANs
(DNS-01), and updates the owner Gateway's listeners.
Defaults stay legacy: `gateway.enabled=false`,
`tenant.spec.gateway=false`. Existing clusters upgrade unchanged.
## External IP allocation — mechanism-agnostic
The per-tenant Gateway's auto-created `LoadBalancer` Service draws its
IP from whatever LB allocator the cluster admin has configured at the
platform layer — same shape as ingress-nginx today. The tenant API stays
mechanism-agnostic — no `gatewayIP` field, no allocator-specific
manifest in the tenant chart. Cozystack itself ships MetalLB installed
but does not render any `IPAddressPool` / `L2Advertisement` /
`BGPAdvertisement` from this chart; admins set up the allocator that
suits their environment (MetalLB pool with L2 / BGP, Cilium LB-IPAM with
announcer, robotlb against a cloud provider, or
`Service.spec.externalIPs` pinning).
If a tenant needs a specific address (DNS already pinned, firewall rule,
etc.), the operator pre-allocates it on the admin side: either
pre-create the Service with `loadBalancerIP` set, or hand the tenant a
reference to a named admin-managed pool. Per-Service IP uniqueness is
the allocator's responsibility — same as for any other LoadBalancer
Service.
## Cert mode: HTTP-01 (default) vs DNS-01 (opt-in)
`publishing.certificates.solver` selects how the controller sources TLS
certs.
**HTTP-01 (default)** — per-app HTTPS listener + per-app Certificate.
New apps require zero platform-side config: deploy the HTTPRoute,
controller does the rest. Inheriting children's hostnames pick up
per-listener certs the same way the owner's own apps do.
**DNS-01 (opt-in)** — single wildcard Certificate (`<apex>` +
`*.<apex>`) covering every published app under the apex, plus per-child
SANs (`<child-apex>` + `*.<child-apex>`) for every inheriting tenant.
Pick this for clusters where many apps share the apex and Let's Encrypt
rate limits matter. The DNS provider account must be able to write TXT
records under every apex level the parent serves — for deeply-nested
inherited children that requires zone delegation or a wide-scope
provider credential.
| `publishing.certificates.dns01.provider` | Required
`publishing.certificates.dns01.<provider>` keys |
| --- | --- |
| `cloudflare` (default) | `cloudflare.secretName`,
`cloudflare.secretKey` |
| `route53` | `route53.region`, `route53.secretName` (and `accessKeyID`
if not on IRSA) |
| `digitalocean` | `digitalocean.secretName` |
| `rfc2136` | `rfc2136.nameserver`, `rfc2136.tsigKeyName`,
`rfc2136.secretName` |
The platform chart wires every provider's keys into `_cluster.dns01-*`;
the per-tenant gateway chart and the cluster-wide `cluster-issuers.yaml`
both read them.
## Strengthens admission coverage on apps.cozystack.io/*
The `cozystack-tenant-host-policy` probe in `gateway.bats` initially
looked like a flake — the apply that was supposed to be VAP-rejected
just succeeded. Investigation traced it to the custom REST handler at
`pkg/registry/apps/application/rest.go`. genericapiserver hands every
storage method a callback into the validating admission chain:
`createValidation` for Create, `updateValidation` for Update,
`deleteValidation` for Delete. `Update` was correctly invoking its
callback since 23e399bd, but `Create` and `Delete` accepted the
parameter and never called it — admission was silently bypassed for both
verbs on every `apps.cozystack.io` resource (Tenant, MariaDB, Postgres,
Kubernetes, …).
This PR wires the missing `createValidation` and `deleteValidation`
calls into Create and Delete. Every ValidatingAdmissionPolicy and
ValidatingWebhook against `apps.cozystack.io/*` now fires on all three
verbs as the API contract requires. Per KEP-3488 aggregated APIServers
are responsible for enforcing admission on their own kinds;
cozystack-api is configured to do so via
`genericoptions.NewRecommendedOptions`, the only thing missing was the
explicit `*Validation` callbacks in the custom REST.
## Security — five guards, grouped by what they defend
Tenants in Cozystack interact with the platform exclusively through
`apps.cozystack.io/*` resources (Tenant, Bucket, Kubernetes, …) served
by `cozystack-api`. Tenant RBAC (`cozy:tenant:*` aggregated to a
RoleBinding in the tenant's own namespace) does not grant write access
to `gateway.networking.k8s.io/*`, core `Namespaces`, or
`cozystack.io/Package`. The protections below split into three groups by
who they defend against:
**Tenant-user-input gates.** Layer 4 (`cozystack-tenant-host-policy`).
`Tenant.spec.host` is the user-supplied field that surfaces as a
security boundary at the hostname layer; gated on every Create / Update
via `cozystack-api`'s admission chain.
**Defense-in-depth.** Layers 1, 2, 5, 7. These do not protect against
tenant-user input (tenants don't hold the relevant RBAC). They guard
against bugs in cozystack-controller / Flux, supply-chain compromise of
an app chart that emits Gateway API resources, and confused-deputy
mistakes by a cluster admin. Fail-closed via `failurePolicy: Fail` +
`validationActions: [Deny]`.
The five layers themselves:
1. **Listener `allowedRoutes` label selector** —
`namespace.cozystack.io/gateway = <owner-tenant-name>`
(controller-written for cozy-* via `AttachedNamespaces`, Helm-written
for tenant namespaces via the tenant chart, both feeding into the same
Gateway selector).
2. **`cozystack-gateway-hostname-policy`** — VAP on `Gateway`
CREATE/UPDATE. Restricts listener hostnames to the namespace's
`namespace.cozystack.io/host` apex.
4. **`cozystack-tenant-host-policy`** — VAP on `Tenant` CREATE/UPDATE.
5. **`cozystack-namespace-host-label-policy`** — VAP on core `Namespace`
CREATE/UPDATE. Immutability gate on the host label.
7. **`cozystack-route-hostname-policy`** — VAP on `HTTPRoute` and
`TLSRoute` CREATE/UPDATE. Closes the hostname-hijack vector across
namespaces under inheritance — a child cannot claim a hostname outside
its own apex.
The previous Layer 3 (`cozystack-gateway-attached-namespaces-policy`)
and Layer 6 (render-time `fail` in cozystack-basics) banned `tenant-*`
entries in `publishing.gateway.attachedNamespaces` on the assumption
that the attach list was the hijack vector. Under inheritance the attach
surface is the label selector, not the static list; the hijack vector is
closed by Layers 4/5/7 independently. Both Layer 3 and Layer 6 are
dropped — kept in numbering above for continuity with the earlier review
threads.
## Foreign-takeover guards
Five reconcile paths refuse to silently rewrite a pre-existing object
that shares the controller-derived name but carries no `OwnerReference`
back to the TenantGateway: `Gateway`, redirect `HTTPRoute`, per-tenant
`Issuer`, wildcard `Certificate`, per-listener `Certificate`. An
operator who hand-pinned a Certificate or Issuer at the controller's
derived name (private CA, manual cert pinning) gets an explicit
`Ready=False/ReconcileError` condition instead of having their config
silently destroyed and the resource re-issued from a different ACME
account.
The namespace-label patching path applies the same discipline:
cozystack-controller only writes / strips
`namespace.cozystack.io/gateway` on namespaces it annotates with
`cozystack.io/gateway-attached-by`. Labels written by the apps/tenant
chart (no annotation) are never touched, so inheritance for tenant
namespaces survives every reconcile.
## Tests
- **Go controller tests** — 50+ cases in
`internal/controller/tenantgateway/`: idempotency, OwnerReference
cascade, multi-parentRef status, HTTP-01↔DNS-01 mode-transition cert
lifecycle, foreign-takeover refusals (Gateway / HTTPRoute / Issuer /
Certificate / per-listener Certificate), DNS-01 provider matrix
(cloudflare / route53 / digitalocean / rfc2136), HTTPS Kinds=[HTTPRoute]
restriction, fail-closed CEL on missing host label, status condition
wiring. Plus, for inheritance: label-based allowedRoutes selector shape,
namespace label patch + garbage-collect via owner annotation, Helm-owned
label safety, DNS-01 wildcard SAN expansion per inheriting child,
per-child wildcard listener materialisation.
- **Admission-chain tests** —
`pkg/registry/apps/application/rest_admission_test.go` pins
`createValidation`, `updateValidation`, `deleteValidation` paths via
sentinel callbacks.
- **helm-unittest** across `extra/gateway` (TenantGateway CR rendering,
both modes), `core/platform` (DNS-01 wiring), `apps/tenant`
(gateway-effective decision matrix including inheritance),
`system/cozystack-basics` (three VAPs + Layer 7 VAP + tenant-root
namespace label including the conditional
`namespace.cozystack.io/gateway: tenant-root` + cozystack-values-secret
`_namespace.gateway` propagation), `system/cert-manager-issuers` (full
DNS-01 provider matrix), `system/cozystack-api` and `system/dashboard`
(Ingress↔TLSRoute/HTTPRoute toggle), `system/monitoring` (Grafana /
Alerta Ingress↔HTTPRoute toggle).
- **e2e bats** — `hack/e2e-apps/gateway.bats` (17 cases) covering
GatewayClass Accepted, Gateway Programmed + LB Service, HTTPRoute
Accepted with status, VAP rejects cross-tenant hostname hijack, VAP
rejects `tenant.spec.host` writes from non-trusted SA
(regression-detector for the admission-chain fix), VAP rejects
`namespace.cozystack.io/host` label changes from non-trusted SA,
route-hostname VAP rejects HTTPRoute hostnames outside the apex, and
three new cases for inheritance — Package admission accepts `tenant-*`
in `attachedNamespaces` (regression guard for the dropped ban), child
tenant without explicit gateway inherits `_namespace.gateway` from a
Gateway-owning parent in lockstep across the namespace label and the
cozystack-values Secret, and the cross-namespace attach end-to-end where
a child tenant's HTTPRoute reaches Accepted on the parent's Gateway via
the inheritance label.
- **CI wiring** — `make test-controllers` target invoked from
`pull-requests.yaml` runs `go test ./internal/...`.
## Backward compatibility
The only always-on change is the flip of `cilium.envoy.enabled` +
`cilium.gatewayAPI.enabled` to `true` (extra `cilium-envoy` DaemonSet,
~100 MB RAM/node idle). Everything else is gated behind
`gateway.enabled=false` / `tenant.spec.gateway=false`. Existing clusters
see no behavioural change until an operator explicitly opts in.
The cozystack-api admission-chain fix is unconditional. Existing
operators who do not define any VAP / admission webhook on
`apps.cozystack.io` resources see no observable change. Operators who DO
have their own webhooks/VAPs against these kinds will start seeing them
fire on Create and Delete (as intended) — review your policies before
merging if you maintain custom admission for `apps.cozystack.io/*`.
Operators who set `publishing.gateway.attachedNamespaces` to include
`tenant-*` entries used to hit a render-time fail or an admission
rejection — now the apply succeeds and the listed tenant namespaces
simply pick up the gateway-attach label alongside the cozy-* system
namespaces. This matches the inheritance model and was kvaps' explicit
ask in the May 25 review.
## Release note
```release-note
- Adds Gateway API support via Cilium, materialised per-tenant via the new `gateway.cozystack.io/v1alpha1 TenantGateway` CRD reconciled by `cozystack-controller`. Opt-in via `publishing.gateway.enabled=true` at the platform level; per-tenant opt-in via `tenant.spec.gateway=true` for dedicated Gateway / LB IP, otherwise the tenant inherits the nearest ancestor's Gateway via a label-based `allowedRoutes` selector (same shape as the existing `_namespace.ingress` inheritance). Includes HTTP-01 (default) and DNS-01 (opt-in, with cloudflare / route53 / digitalocean / rfc2136 providers) cert solvers; DNS-01 extends the parent's wildcard Certificate with per-child apex SANs and adds a `*.<child-apex>` listener per inheriting tenant. Tenant API is mechanism-agnostic for IP allocation — the LoadBalancer Service draws its IP from the cluster's admin-configured allocator (MetalLB / Cilium LB-IPAM / robotlb / externalIPs).
- Strengthens admission coverage on `apps.cozystack.io/*` resources: `cozystack-api` now invokes `createValidation` and `deleteValidation` callbacks on Create and Delete (Update was already wired). Operators with custom ValidatingAdmissionPolicies or webhooks targeting `apps.cozystack.io/*` will see them fire on all three verbs as the API contract requires; review your policies before merging if you maintain such admission.
```
## What this PR does
Concurrent PR builds were racing in the OCIR registry. Every package
Makefile tagged its images via `settag`, which on any non-release commit
collapsed to `:latest` — so every image across every PR was pushed to
the same tag, and parallel builds got 409s on the manifest PUT. Two
outliers (`metallb`, `redis-operator`) used hardcoded version tags and
collided across PRs unconditionally.
This PR replaces `settag` and the `TAG=latest` fallback in
`hack/common-envs.mk` with a single `image-tags` macro driven by three
env vars:
| Env | Meaning |
|---|---|
| `IMAGE_TAG` | Build-unique tag, always pushed |
| `PUBLISH_VERSIONED` | Also push `:<component-version>` (when provided)
|
| `PUBLISH_FLOATING` | Also push `:latest` |
Wired into the two workflows:
- **`pull-requests.yaml`** — `IMAGE_TAG=pr-<N>-<sha>`, both `PUBLISH_*`
off. Every PR build pushes a unique tag set; no collisions, no `:latest`
move.
- **`tags.yaml`** — `IMAGE_TAG=<ref_name>`, both `PUBLISH_*` on. Release
output preserved (component versions + `:latest`) plus a uniform
`:<git-tag>` handle on every image for traceability.
- **Local** — defaults to `IMAGE_TAG=dev`, no flags, `PUSH=0`.
All 30 package Makefiles converted to `$(call
image-tags,<repo>,<versioned-tag>)`. Image consumers reference digests
pinned into `values.yaml` by each Makefile, so the tag is decorative for
cluster-side pulls — the build-unique tag is sufficient for E2E.
Also drops the redundant `:<component-version>-<git-tag>` form (cilium,
linstor, matchbox, ubuntu-container-disk, kubevirt-*,
cluster-autoscaler). Nothing in the repo or downstream values.yaml
consumed it; the same information is preserved via the new `:<git-tag>`
plus the existing `:<component-version>` tags on release.
`ubuntu-container-disk` builds a separate image per Kubernetes version
in a loop, so its primary tag became `:<k8s-ver>-<IMAGE_TAG>` to prevent
loop iterations within a single build from pushing to the same tag.
Verified by dry-running `make -n image` across representative packages
(simple, dual-tag, skopeo, flux-push, loop) under all four contexts (PR
/ release / main / dev) — tag sets are unique and behave as expected.
### Release note
```release-note
refactor(build): PR CI now pushes images under a unique `pr-<N>-<sha>` tag per build instead of `:latest`. Concurrent PR builds no longer race in the registry. Release builds keep the existing `:<component-version>` and `:latest` tags and additionally publish a uniform `:<git-tag>` handle across every image. The redundant `:<component-version>-<git-tag>` tag (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) is no longer published — image references in values.yaml are digest-pinned and were the only legitimate consumers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Centralized and simplified container image tagging across build
workflows.
* PR builds now use unique image tags derived from PR metadata for
better isolation.
* Added configurable flags to control publishing of versioned and
floating tags, making release tagging behavior more predictable.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2711?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
v1.5.0 adds the @immutable annotation support
(cozystack/cozyvalues-gen#24). Without the bump, pre-commit re-runs
make generate with the v1.4.0 binary, silently strips any
x-kubernetes-validations entries that downstream PRs add to
values.schema.json / embedded openAPISchema, and reports drift against
the committed artefacts.
This is a no-op for the current main: no chart currently uses
@immutable, so regeneration with v1.5.0 produces identical output to
v1.4.0. Unblocks cozystack/cozystack#2639 (apply @immutable to
storageClass across stateful apps).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The initial Scorecard workflow used floating major-version tags
(@v2, @v4, @v3). ossf/scorecard-action does not publish a floating
v2 tag — only specific patch releases — so the first scheduled run
failed with `Unable to resolve action ossf/scorecard-action@v2`.
Pin every action by its full commit SHA with a trailing comment
showing the human-readable version. This also satisfies the
`Pinned-Dependencies` Scorecard check itself.
- actions/checkout@de0fac2 (v6.0.2)
- ossf/scorecard-action@4eaacf0 (v2.4.3)
- actions/upload-artifact@043fb46 (v7.0.1)
- github/codeql-action/upload-sarif@03e4368 (v3)
Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io>
Runs the OpenSSF Scorecard action on:
- weekly schedule (Mon 06:00 UTC)
- branch protection rule changes
- pushes to main
- manual dispatch
Publishes results to:
- the public OpenSSF Scorecard API (scorecard.dev) for transparency
- the GitHub code-scanning dashboard (Security tab) for triage
No README badge is added yet — the badge will be introduced in a
follow-up PR once the first scans have run and any quick-win
improvements (token permissions, dependency pinning) have landed.
Aligns with the project's CNCF Incubation readiness work and the
security self-assessment in progress.
Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io>
v1.4.0 adds the `intOrString` type alias used by the postgres package's
`postgresql.parameters` schema in this PR. CI must install v1.4.0 so the
"Verify generated code is up to date" check regenerates the same schema
shape (anyOf + x-kubernetes-int-or-string: true) rather than reverting
to {type: string}.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`release_next.sh` and `init_version.sh` (via `update-all → init-version`)
create `data/versions/<DOC_VERSION>.yaml` for new versions, but the
`update-website-docs` job only staged `content` and `hugo.yaml`. The new
file was silently dropped from the website PR, breaking
`{{< version-pin >}}` lookups for the released minor.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace MAINTAINERS.md snapshot with a list derived from actual recent
activity on main (commits, merged PRs and reviews over the last three
months). MAINTAINERS.md had drifted in both directions — listing people
no longer active on the project while missing several current
maintainers — so copying it verbatim was both handing SSH access to
inactive accounts and locking out people who would actually use this
debug feature.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
When the e2e job fails on a PR carrying the `debug` or `release` label
(release PRs are bot-created, so the label implies maintainer authorship),
hold the runner and open an SSH breakpoint via cozystack/breakpoint-action
(a fork of namespacelabs/breakpoint-action pinned by SHA) so a maintainer
can attach, inspect Talos/Cozystack state, and resume.
The fork adds three things on top of upstream:
* `mode: pause-idle` — waits 20m for the first SSH connection, then keeps
the workflow paused while sessions are active, exiting 10m after the
last session disconnects. Saves runner-minutes vs a fixed timeout.
* `endpoint` step output + `::notice::` annotation — surfaces the SSH
endpoint while the step is still paused (step logs are buffered until
step completion, so the endpoint would otherwise be invisible).
* A dedicated Check Run "Breakpoint Open" with conclusion=failure
carrying the SSH endpoint and sandbox-entry hint in output.summary
(Markdown, rendered in the UI). Surfaces as a standard ✗ in
`gh pr checks` so any CI watcher sees the open breakpoint without
bespoke marker parsing. Updated to conclusion=success when the
breakpoint exits.
The step is configured via the `BREAKPOINT_ENDPOINT` repo variable. In
fork PRs, repo variables are not exposed, so the step is naturally
skipped and forks cannot reach the self-hosted rendezvous server. The
`authorized-users` list is kept as a second defense layer at the
breakpoint level.
Permissions on the e2e job: `checks: write` (for Check Run create+update).
Upstream PR: namespacelabs/breakpoint-action#16
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Mark the generate-changelog job with continue-on-error and switch the
update-website-docs gate to always() so a failed or skipped changelog
generation no longer blocks the website docs update. prepare-release
remains the hard gate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The 2-hour ceiling on the E2E job was clipping otherwise-working
runs on the oracle-vm-24cpu-96gb-x86-64 runners: image pulls,
multi-stage Talos bootstrap, full helm-release fan-out plus the
post-install bats matrix routinely consume 100-180 minutes of
wall-clock, and runs that would have eventually gone green were
getting cancelled by GitHub at the timeout.
Recent successful E2E runs on the same runner pool sit between
60 and 184 minutes; lift the cap to 180m to cover the long tail
without inviting truly-stuck jobs to linger forever.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
Implemented CNPG-strategy controller for postgres backups with supporting in-place and to-copy restores.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* CNPG-based backup strategy for PostgreSQL with S3-backed object store
and controller support
* Option to reference existing Kubernetes Secret for S3 credentials
(configurable key names)
* Controller supports scheduled/ad-hoc backups and both in-place and
to-copy restores; safer credential handling (no inline keys)
* **Examples**
* Complete manifests: bucket, credentials, CNPG strategy, BackupClass,
Plan, BackupJob, RestoreJob, Postgres source/target
* **Security/RBAC**
* Controller RBAC expanded for CNPG resources; tests enforce no
cluster-wide Secret read
* **Documentation**
* README, chart values/schema and UI schema updated to document
secret-reference workflow and restore guidance
* **Tests**
* New end-to-end and unit tests covering CNPG flows, restore semantics,
cleanup behavior and RBAC constraints
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Two compounding issues caused merging a backport PR to spawn another
backport (e.g. #2581 [Backport release-1.2] [Backport release-1.3] ...
from #2579):
1. pr-labeler auto-added the `backport` label to any PR whose title
started with `[Backport release-X.Y]` (regression introduced in
c8ed1c65 when pr-labeler.yaml was first added). korthout/backport-
action puts that prefix on every backport PR it creates, so the
label was being re-applied to every backport, satisfying the
workflow's gate when the backport itself was merged.
2. Even with that fixed, an external labeling bot (dosubot) still
re-applies `backport-previous` to backport PRs based on similarity
to the parent. That bot is configured at the org level and cannot
be controlled from this repo.
Drop the `toAdd.add('backport')` line from pr-labeler so the label
isn't propagated by us, and gate both backport jobs on
`pull_request.base.ref == 'main'` so backport PRs (which target
release-X.Y) cannot trigger the workflow regardless of what labels
end up on them.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Replace Dosu's stale-management and PR-size labeling with native GitHub
Actions ahead of disabling the Dosu app on this repo, and migrate the
six `size:*` labels to the canonical `size/*` namespace while we're
rewriting the sizing path.
**Background**: Dosu (the SaaS) currently writes `lifecycle/stale`
lifecycle and `size:*` labels via its bot. Audit of `.github/workflows/`
(full grep) confirmed zero hard dependencies on Dosu's outputs —
`backport*` is human-applied, `release` is auto-applied by `tags.yaml`
itself, and nothing else consumes Dosu-applied labels in any workflow.
### `.github/workflows/stale.yaml` (new)
Daily cron at 04:37 UTC plus manual dispatch. Uses `actions/stale@v10`:
- Stale after 60 days idle: applies `lifecycle/stale`.
- Closes 14 days later, applying `lifecycle/rotten`.
- `remove-stale-when-updated: true` — activity un-stales automatically.
- 100 ops/run.
Issue and PR exempt lists differ on purpose: an accepted issue
(`triage/accepted`) is a known long-tail task, not stale; a held PR
(`do-not-merge/hold`) is paused intentionally.
Priority/security/regression labels protect issues someone explicitly
decided to keep visible.
### `.github/workflows/pr-size.yaml` (new)
Triggers on `opened`, `synchronize`, `reopened` (`pull_request_target`).
Inline `actions/github-script@v7` — no third-party labeler dependency.
Diff-line count with thresholds matching the descriptions in
`.github/labels.yml`:
| label | lines |
|---|---|
| `size/XS` | 0–9 |
| `size/S` | 10–29 |
| `size/M` | 30–99 |
| `size/L` | 100–499 |
| `size/XL` | 500–999 |
| `size/XXL` | 1000+ |
Inline ignore patterns skip:
- vendored Go deps (`vendor/**`)
- kubebuilder/openapi inline generated files
(`zz_generated.{deepcopy,conversion,defaults,openapi}.go`)
- whole code-generated trees: any path under `generated/`, plus
`*.pb.go`
- vendored Helm charts (`packages/system/*/charts/**`)
- lockfiles (`go.sum`, `*.lock`, `*.lockb`)
Concurrency group keyed on PR number with `cancel-in-progress: true`
(latest push wins). `removeLabel` is wrapped in `try/catch` tolerating
404 for races between event payload and execution.
### `size:` → `size/` namespace migration
The legacy `size:X` (colon) form was an artefact of the previous sizing
bot; with that bot being replaced and every other label namespace in the
repo using slashes (`kind/`, `area/`, `priority/`, `triage/`,
`lifecycle/`, `do-not-merge/`, `security/`), `size/` is the consistent
form. Migration via aliases in `labels.yml` — IDs preserved, every
existing labelled PR follows the rename automatically.
The new `pr-size.yaml` emits `size/*` from day 1 and accepts both
prefixes when looking up the previous size label, to handle the brief
window between merge and the next labels-sync run.
### Out of scope (intentional)
- **`lgtm` replacement** — deferred. 33/33 of `lgtm` events in the last
100 PRs come from Dosu; humans never apply it manually. Whether
reviewers actually rely on it is unclear — observe for 7 days
post-Dosu-disable before adding a `pull_request_review`-on-approve
workflow.
- **LLM-based issue auto-classification** (`kind/bug`/`kind/feature` on
inbound issues) — accepted loss; manual triage replaces.
- **Cleanup of `wontfix`/`invalid`** in `labels.yml` — separate cleanup
PR.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
- Pull requests now automatically receive size labels (XS, S, M, L, XL,
XXL) based on lines changed.
- Issues and pull requests are automatically marked as stale after 60
days of inactivity and closed 14 days later if still inactive.
* **Chores**
- PR size label namespace updated to a slash-based format (size/...)
while preserving the old names as aliases for backward compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `generate-changelog` job in `.github/workflows/tags.yaml` previously
checked out `main` and ran the AI agent there. The agent followed
`docs/agents/changelog.md`, which instructed it to compute the release
range with `git log <previous_version>..HEAD`. That works for minor
releases (cut from `main`), but it breaks for any patch release cut from
a `release-X.Y` branch — `HEAD-on-main` is a strict superset of the tag
and contains commits that were merged to `main` both before and after
the tag.
The v1.3.1 changelog generated by this workflow (#2480) demonstrated the
failure mode: 8 PRs that were merged to `main` but never shipped in
v1.3.1, 6 backport PRs that landed on `release-1.3` *after* v1.3.1 was
tagged, both originals and their backports as separate entries, a
hallucinated 2024 PR (#435), and the `cozystack-ci` bot in the
contributors list. The corrected v1.3.1 changelog is in #2480.
This PR fixes the root cause and tightens the agent guardrails:
* **`tags.yaml`** — check out the release tag commit (`ref: ${{
steps.tag.outputs.tag }}`) instead of `main`, so `HEAD == release
commit` and `git log v<prev>..HEAD` corresponds to what the tag actually
contains. The "Create changelog branch" step still creates the PR branch
from `origin/main`, so PRs continue to merge cleanly.
* **`tags.yaml`** — the AI prompt now states explicitly that `HEAD` is
the release commit and that the upper bound of the range is the new tag,
never `main`.
* **`docs/agents/changelog.md`** — every example replaces `..HEAD` with
`..v<new_version>`, so the instruction is unambiguous regardless of
which branch is checked out.
* **`docs/agents/changelog.md`** — hard rule: backport PRs MUST be
combined with the original into a single entry (`#1606, backport
#1609`), never listed as a second entry. Documented edge case: if the
original isn't in the range, drop the entry entirely (it shipped in a
previous release).
* **`docs/agents/changelog.md`** — forbid using the brief description
verbatim as the detailed description (the failure was `* **fix(foo):
X**: fix(foo): X (...)`), and forbid inventing entries for PRs outside
the release range.
* **`docs/agents/changelog.md`** — filter bot/CI accounts (`app/*`,
`*[bot]`, `cozystack-ci`, `github-actions`, `dependabot`, `renovate`)
out of the human Contributors list. Bot attribution on individual
entries is still valid.
The v1.3.1 changelog correction itself lives in #2480 (separate scope).
### Release note
```release-note
fix(ci): tags.yaml now generates the release changelog from the tag commit instead of `main`, fixing patch-release changelogs (e.g. v1.3.1) that previously included unrelated commits from `main`. The AI prompt and `docs/agents/changelog.md` were tightened to forbid duplicate backport entries, title-as-description, fabricated PRs, and bot accounts in the Contributors list.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Changelog generation now checks out the release tag commit and
verifies whether the release changelog already exists on main before
generating.
* Updated prompts and tooling to treat the release tag as the upper
bound and avoid using HEAD when computing diffs.
* Enforced combining original and backport PRs into single changelog
entries with explicit “backport #…”.
* **Documentation**
* Strengthened authoring rules (distinct brief/detail, forbid invented
entries, PRs must fall within tag-to-tag ranges).
* Updated examples and guidance to use tag-to-tag comparisons and
improved contributor extraction to use bold @user attribution while
excluding automation accounts and the current release file.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- tags.yaml: the check_changelog step now looks at origin/main rather than
the working tree. After Round 2 we check out the release tag, which never
has the changelog file (the PR opened by this job merges to main after
the tag is cut), so the previous `[ -f ]` test always returned false and
the gate was effectively dead on reruns.
- changelog.md: Step 6's `RELEASE_END=$(git log -1 --format=%ai HEAD)`
(lines 179, 222) was missed in the previous "never HEAD" sweep — now
uses v<new_version>, matching Step 5.
- changelog.md: Step 2 ("on main branch") was incompatible with the CI flow
which now runs detached at the release tag. Relaxed it to list both
configurations (interactive on main; CI detached at v<version>), while
the "Scope and boundaries" section continues to forbid the agent from
switching branches.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- tags.yaml: drop "(or HEAD)" from the changelog prompt; the doc says
"never HEAD" and the workflow now matches.
- docs/agents/changelog.md: exclude the current release file from the
"previous contributors" set used for first-time-contributor detection.
The previous globbing on `docs/changelogs/v*.md` swept up the file
being generated, so `comm -23` would never report any newcomers.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Publish draft release step used the workflow's default GITHUB_TOKEN,
which does not return draft releases from listReleases — so the script
failed with "Draft release for vX.Y.Z not found" and every release tag
had to be undrafted manually.
This same bug was fixed once before in 4db55ac5eb ([ci] Add Github
token to fetch draft releases) by adding github-token: GH_PAT to the
listing step. It was lost during the refactor in 66a756b6 that combined
the separate listing/flags/publish steps into a single Publish step,
and not restored when f96b2da1 migrated remaining github-token usages
from GH_PAT to the cozystack-ci App token.
Add github-token: app-token.outputs.token to the merged step so it
shares the same auth as the rest of the workflow.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The generate-changelog job in .github/workflows/tags.yaml previously
checked out main and ran the AI agent there. The agent followed
docs/agents/changelog.md, which instructed it to compute the release
range with `git log <previous_version>..HEAD`. For minor releases
(vX.Y.0) cut from main this happens to give the right answer; for
patch releases (vX.Y.Z, Z>0) cut from a release-X.Y branch, HEAD-on-
main is a strict superset of the release tag and contains commits
that were merged to main both before and after the tag.
The v1.3.1 changelog generated by this workflow (#2480) demonstrated
the failure mode: it included 8 PRs merged to main but never shipped
in v1.3.1, 6 backport PRs that landed on release-1.3 *after* v1.3.1
was tagged, both originals and their backports as separate entries,
a hallucinated 2024 PR, and the cozystack-ci bot in the contributors
list. The corrected v1.3.1 changelog contains the single fix that
v1.3.1 actually ships.
This PR fixes the root cause and tightens the agent guardrails:
* tags.yaml: check out the release tag commit (`ref: ${{ steps.tag.
outputs.tag }}`) instead of `main`, so HEAD == the release commit
and `git log v<prev>..HEAD` corresponds to the actual release. The
Create-changelog-branch step still creates the PR branch from
origin/main, so PRs continue to merge cleanly into main.
* tags.yaml: update the AI prompt to state explicitly that HEAD is
the release commit and that the upper bound for the release range
is the new tag, not main.
* docs/agents/changelog.md: replace `..HEAD` with `..v<new_version>`
in every example so the instruction is unambiguous regardless of
which branch is checked out.
* docs/agents/changelog.md: add a hard rule that backport PRs MUST
be merged into the original entry (single line, two PR numbers),
never listed as a second entry. Document the edge case where the
original PR is outside the range — drop the entry.
* docs/agents/changelog.md: forbid using the brief description
verbatim as the detailed description, and forbid inventing entries
for PRs that are not in the release range.
* docs/agents/changelog.md: filter bot/CI accounts (`app/*`,
`*[bot]`, cozystack-ci, github-actions, dependabot, renovate) out
of the human Contributors list. Bot attribution on individual
entries is still valid.
The corresponding correction for the v1.3.1 changelog itself is in
PR #2480 on the changelog-v1.3.1 branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
- Adds a pre-commit hook and a dedicated GitHub Actions workflow that
run `make generate` at the repo root to catch drift in generated API
code (CRDs, DeepCopy, clients, RBAC) before it lands on main.
- Pre-commit hook is scoped to paths that actually affect codegen
(`api/`, `pkg/apis/`, `hack/update-codegen.sh`,
`hack/boilerplate.go.txt`) so unrelated commits are not slowed down.
- CI workflow (`.github/workflows/codegen-drift.yml`) installs Go from
`go.mod`, runs `make generate`, and fails on drift with an error
pointing contributors to the local fix.
- Includes one drift fix the check surfaced: `RestoreJobSpec.Options`
(added in #2437) had no `DeepCopyInto` handling — regenerated.
### Release note
```release-note
ci(api): add pre-commit hook and GitHub Actions workflow that verify generated API code (CRDs, deepcopy, clients, RBAC) is in sync with `make generate`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added a CI workflow that checks generated code during PRs and fails
the build if generated artifacts diverge.
* Added a pre-commit hook that runs generation checks locally to prevent
committing outdated generated files.
* Fixed deep-copy behavior for backup restore job specs so nested
options are correctly duplicated.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from coderabbitai on .github/workflows/stale.yaml:37:
add priority/important-longterm to exempt-issue-labels. The label is
documented in labels.yml as "Important over the long term, but may
not be staffed and/or may need multiple releases to complete" — such
issues sit idle for 60+ days by design and must not be auto-staled.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add two GitHub-native workflows ahead of disabling the Dosu app and
migrate size labels to canonical k8s namespacing.
- .github/workflows/stale.yaml uses actions/stale@v10 to manage the
lifecycle/stale and lifecycle/rotten lifecycle. Daily cron, 60d
before stale, 14d before close. Issue/PR exempt lists differ on
purpose: an accepted issue (triage/accepted) is a known long-tail
task, not stale; a held PR (do-not-merge/hold) is paused
intentionally. Priority/security/regression labels protect issues
someone explicitly decided to keep visible.
- .github/workflows/pr-size.yaml uses actions/github-script@v7 to
apply size/* labels based on diff line count. Thresholds match the
descriptions in .github/labels.yml. Inline ignore patterns skip
vendored deps, kubebuilder/openapi inline generated files
(zz_generated.{deepcopy,conversion,defaults,openapi}.go), whole
code-generated trees (any path under generated/ or *.pb.go),
vendored Helm charts, and lockfiles. Concurrency group keyed on PR
number with cancel-in-progress: true. removeLabel is wrapped in
try/catch tolerating 404 for races between event payload and
execution.
- Migrate the six size labels from legacy 'size:X' (colon) to
canonical 'size/X' (slash) via aliases in .github/labels.yml. The
colon form was an artefact of the previous sizing bot's output;
with the bot replaced and namespacing already used everywhere else
(kind/, area/, priority/, triage/, lifecycle/, do-not-merge/,
security/), the slash form aligns with the rest of the scheme. The
alias migration preserves IDs and keeps every existing labelled PR
pointing at the renamed entry.
pr-size.yaml emits the new 'size/' form from day 1 and accepts
either prefix when looking up the previous size label, to handle
the brief window between PR merge and the labels-sync run.
Verification of zero hard dependencies on Dosu in workflows: only
backport.yaml (consumes backport*, applied by humans only) and
pull-requests*.yaml + tags.yaml (consume release, applied by tags
workflow itself) — none of those labels come from Dosu.
The lgtm label (33/33 in last 100 PRs from Dosu, none from humans)
is intentionally not replaced in this PR — observe reviewer
behaviour for 7 days after Dosu is disabled and decide whether a
pull_request_review-on-approve workflow is needed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:160,173:
emit core.warning when a Conventional Commits type has no kind/*
mapping or a scope has no area/* mapping. Without the warning, typos
(e.g., "hotfix" instead of "fix") and recurring new scopes silently
fall through to area/uncategorized, masking that the mapping has
drifted.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:155:
Conventional Commits 1.0 spec item 16 treats BREAKING CHANGE: and
BREAKING-CHANGE: as synonymous footers. The hyphen form was silently
ignored before, so PRs that use it would miss kind/breaking-change.
https://www.conventionalcommits.org/en/v1.0.0/#specification
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from myasnikovdaniil on .github/workflows/pr-labeler.yaml:129:
defensive (pr.labels || []) avoids TypeError if the webhook payload
arrives without the labels field on edge cases like stripped edited
events.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add .github/labels.yml as the canonical label set, synced into the
repository by .github/workflows/labels.yaml using EndBug/label-sync.
Conventions follow the Kubernetes scheme:
https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md
Six namespaced groups: kind/, priority/, triage/, lifecycle/, area/,
do-not-merge/. Cozystack-specific labels preserved (epic, community,
security/*, size:*).
Migration via aliases keeps references on existing issues and PRs:
- bug -> kind/bug
- enhancement -> kind/feature
- documentation -> kind/documentation
- question -> kind/support
- frozen -> lifecycle/frozen
- stale -> lifecycle/stale
- do-not-merge -> do-not-merge/work-in-progress
delete-other-labels is false on the initial rollout; redundant labels
("do not merge", duplicate, invalid, wontfix) stay until a follow-up
PR removes them after stabilisation.
The labels workflow has a validate job (python3 schema check) that
runs on PR. Sync runs only on push to main, weekly cron, and manual
dispatch. Schema invariants:
- description <= 100 chars (GitHub REST API limit)
- color is 6-char hex without leading #
- unique top-level names
- aliases do not collide with top-level names
PR auto-labeling (.github/workflows/pr-labeler.yaml):
- Parses PR title as Conventional Commits header (type, scope, !).
- type -> kind/* (feat -> kind/feature, fix -> kind/bug, docs ->
kind/documentation, chore/refactor -> kind/cleanup; style, perf,
test, build, ci, revert -> no kind label).
- scope -> area/* via embedded mapping; composite scopes split on
comma. Bracket-style fallback ([scope] description) maps area/*
but cannot infer kind/*.
- '[Backport release-1.x]' prefix is stripped; area/release and
backport labels are added.
- '!' after type or 'BREAKING CHANGE:' footer in body adds
kind/breaking-change.
- Unmapped scope or non-conventional title adds area/uncategorized
to flag for human review.
- Additive only — never removes existing labels.
Hardcoded label references updated:
- .github/ISSUE_TEMPLATE/bug_report.md (bug -> kind/bug)
- .github/workflows/tags.yaml (documentation -> kind/documentation)
AGENTS.md gains an Activation entry pointing agents to labels.yml
as the source of truth and to contributing.md for the title
auto-labeling table.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Add generated-output dirs (pkg/generated, internal/crdinstall/manifests,
packages/system/*/definitions) and Makefile to the workflow paths: filter
so PRs that modify only generated artifacts still trigger the drift check.
- Mirror the same paths in the root pre-commit hook's files: regex so
manual edits to generated files or changes to the root generate target
re-run make generate through pre-commit.
- Switch drift detection in the workflow from `git diff --exit-code` to
`git status --porcelain` so new untracked files produced by make generate
(e.g. generated YAML/Go for a new API type) also fail the job; dump
`git diff --color=always` on failure for easier debugging.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/update-codegen.sh sources kube_codegen.sh from the Go module
cache at ~/go/pkg/mod/k8s.io/code-generator@vX.Y.Z/, but the module
is not declared in go.mod so a fresh runner has nothing to source
from. Add a workflow step that parses the pinned version out of the
script and pulls the module into the cache before running
make generate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the Generate changelog using AI step ran
Copilot with --allow-all-tools and GH_TOKEN set to the write-capable
installation token issued to the job (contents: write,
pull-requests: write on all cozystack/* repos). The scope rules in
docs/agents/changelog.md and the step prompt tell the agent not to
use those permissions, but nothing at the token layer prevented it.
Mint a second, read-only installation token from the same app
(same COZYSTACK_CI_APP_ID / COZYSTACK_CI_PRIVATE_KEY, scoped to
contents/pull-requests/metadata read) and pass that one to the AI
step instead. The write-capable token is still used by the checkout,
commit/push, and PR-creation steps that actually need it.
This is defense in depth: even if a future prompt change or agent
misbehavior ignored the scope rules, the token itself has no write
capability on any repository in the cozystack org. No new secret,
no new GitHub App install, no admin-side change — the RO token is
minted in the same workflow from the same app credentials.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the existing `[ -f ]` check catches
missing files but a zero-byte `docs/changelogs/v${VERSION}.md` would
still be staged and committed — `git add` + `git commit -s` on a new
empty file succeeds and produces a real commit, leaving the
downstream PR with no actual changelog content.
Add a `[ -s ]` guard after the existence check: if the Generate
changelog using AI step produces an empty file, emit a matching
`::error::` annotation and exit 1 before snapshotting.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three changes to the generate-changelog job to fix the v1.3.0
release pipeline failure (run 24765377017) and make the job robust
to whatever state the Copilot step leaves behind.
1. Add `timeout-minutes: 30` to the Generate changelog using AI
step. On the v1.3.0 re-run the step hung silently for 10+
minutes; with no timeout a hung Copilot would hold a self-hosted
runner for up to 6 hours (job default). The previous successful
run took ~26 minutes, so 30 is a reasonable ceiling.
2. Replace the terse, ambiguous Copilot prompt with a one-liner
that invokes docs/agents/changelog.md directly. The "Scope and
boundaries" section added to that doc in the previous commit is
now the single source of truth for what the agent may and may
not do, so the workflow only needs to pass the version and
point at the relevant doc. VERSION is moved to step env: to
match GitHub's workflow-injection hardening guidance.
3. Rewrite the Create changelog branch and commit step:
- add `set -euo pipefail` so any failure is visible
- validate the file exists up front and fail loud with
`::error::` if not
- copy the file to a tempfile BEFORE `git checkout -b`, so the
checkout to `origin/main` cannot remove it (this is the fix
for the original pathspec error the v1.3.0 run hit)
- use `trap` to clean up the tempfile on any exit path
- move VERSION to env
- drop the dead "no changes to commit" branch: the check_changelog
step earlier in the job gates this step on the file being
absent from origin/main, so `git add` + `git commit` must
produce a diff. If they don't (e.g. Copilot emitted an empty
file), fail loud instead of pushing an empty branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Run root 'make generate' as a pre-commit hook and as a dedicated
CI workflow so missed codegen updates (CRDs, deepcopy, clients, RBAC)
are caught instead of merging stale generated files.
Pre-commit hook is scoped to files that actually affect codegen
(api/, pkg/apis/, hack/update-codegen.sh, hack/boilerplate.go.txt)
so unrelated commits are not slowed down. CI job sets up Go from
go.mod, runs make generate, and fails on drift with a pointer to
the local fix.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The website repo is switching to a permanent `content/en/docs/next/`
trunk (cozystack/website#495). Released version directories are no
longer pre-created — the upstream release workflow is now responsible
for explicitly promoting `next/` → `vX.Y/` on new minor/major releases
via `make release-next`.
Update the `update-website-docs` job to match that contract:
- New `Determine if this release promotes next/` step parses the tag
and only enables promotion for final `vX.Y.Z` tags where
`content/en/docs/vX.Y/` does not already exist. Prereleases and
patch releases keep the old behaviour (target routing is handled
by the website Makefile on its own).
- New `Promote next/ to released version` step runs
`make release-next RELEASE_TAG=...` before `make update-all`, so
the subsequent `update-all` routes into the freshly-created
`vX.Y/` directory and `next/` stays untouched as the trunk for
the following release.
- `git add content hugo.yaml` to pick up the version registration
that `release-next` writes via `register_version.sh`.
- Convert the existing `update-all` step to use `env:` vars,
matching the new step and addressing the workflow-injection
hardening guidance.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The cozystack-ci[bot] noreply email with brackets is rejected by DCO
probot as an invalid email address, causing release PRs to fail DCO
checks.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The GH_PAT secret tied to cozystack-bot is no longer valid after
migrating release workflows to the cozystack-ci GitHub App in #2351.
This broke the resolve_assets and e2e jobs for release PRs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace the cozystack-bot personal access token (GH_PAT) with a
GitHub App (cozystack-ci) installation token across all CI/CD
workflows. This improves security by using short-lived, scoped
tokens instead of a long-lived PAT tied to a user account.
Changes:
- tags.yaml: use app token in all 3 jobs (prepare-release,
generate-changelog, update-website-docs)
- auto-release.yaml: use app token for daily patch releases
- pull-requests-release.yaml: use app token for release finalization
The cozystack-ci GitHub App (ID: 3297617) is installed org-wide
with contents:write and pull-requests:write permissions. Secrets
COZYSTACK_CI_APP_ID and COZYSTACK_CI_PRIVATE_KEY are set at org
level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Always force-create and force-push the API submodule tag so it stays
in sync with the main version tag, even when re-tagging.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Remove broken `git push origin HEAD` in detached HEAD state (commit
artifacts are already pushed via the "Create release branch" step)
- Make subtag push idempotent: use explicit refs/tags/ prefix and
tolerate already-existing remote tags after verifying they point to
the correct commit
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Reduces flakiness in E2E tests by addressing the most common failure
modes.
### Kubernetes tenant tests (`run-kubernetes.sh`)
- Increase node Ready timeout from 2m to 5m — CI runners are shared and
resource-constrained
- Fail fast when nodes are not Ready — saves ~7 minutes per failed
attempt by not running LB/NFS tests that will also fail
- Delete stale Kubernetes resources at test start — retries provision
from scratch instead of patching stuck state
- Wait for port-forward to be ready before using it (fixes race
condition)
- Reduce version check polling interval from 5s to 1s
### All app tests (postgres, redis, kafka, clickhouse, mariadb, mongodb,
qdrant, foundationdb, openbao, vminstance, bucket)
- Add pre-cleanup of stale resources from previous failed retries so
each attempt starts clean
### Test runner (`cozytest.sh`)
- Clean up temp directory on test failure (was only cleaned on success,
leaking `/tmp` dirs)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved e2e robustness by deleting stale resources before creation
(ignore-if-missing, bounded time) and killing leftover port-forwards.
* Added an exit cleanup to consistently remove temporary artifacts and
teardown port-forwards on any exit.
* Added readiness polling for forwarded endpoints (curl with timeout);
made API/version retries more responsive.
* Extended node readiness wait (2m → 5m), dump debug info and fail fast
if unready.
* Made load‑balancer reachability checking explicit and more reliable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Without explicit timeouts, stuck jobs use the GitHub Actions
default of 360 minutes (6 hours). This blocks the concurrency
group and prevents new runs from starting.
Set reasonable limits:
- Build: 30 minutes (normally ~5 min)
- E2E: 120 minutes (normally ~60 min)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pull-requests workflow used paths-ignore at the trigger level, which
prevented the entire workflow from running on docs-only PRs. This meant
the required "E2E Tests" check was never created, blocking merge for
non-admin users.
Replace trigger-level paths-ignore with a detect-changes job using
dorny/paths-filter. The workflow now always triggers (so checks are
always reported), but build and downstream jobs are skipped when only
docs files change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add -talos suffix to the default variant output file for consistency
with -generic and -hosted variants. Update all references in CI
workflows, e2e tests, upload scripts, and testing Makefile.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Aleksei Sviridkin
