The lineage-controller-webhook now sets a
scheduler.cozystack.io/scheduling-class label on mutated pods (in
addition to the existing annotation of the same name).
This unlocks SchedulingClass authors writing podAffinity rules whose
labelSelector targets pods across different applications that share
the same class. A natural pattern becomes: define a class whose name
is the affinity-group identifier, attach it to several applications,
and have the scheduler co-locate them at the chosen topology key
(region, zone, hostname).
Behavior preserved when the SchedulingClass CR is missing: pods are
left untouched (no scheduler injection, no label) so they do not stay
Pending on a non-existent scheduler.
Tests cover label/annotation/schedulerName injection, existing label
preservation, non-Pod no-op, no-class no-op, and missing-CR no-op.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Every admission walks the owner chain (Pod, StatefulSet, HelmRelease,
application CR) with one uncached GET per level, and sibling Pods
repeat the same lookups. On a cluster with many tenants that put the
lineage webhook at 300-500ms p99, the largest share of apiserver
admission latency.
Owners are now reused for 60 seconds, successful lookups only. A failed
lookup, whether RESTMapping, NotFound or a timeout, leaves the walk
without an owner, so the object is admitted with no lineage labels and
stays unlabeled until its next UPDATE runs the walk again. Secrets,
Services and PVCs can go a long time between updates, so a cached
failure would leave every object admitted within the TTL unlabeled for
that long, and the aggregated cozystack-api makes such failures
routine: discovery of apps.cozystack.io fails while it rolls.
Non-Pod objects that already carry every lineage label, tenantresource
included, skip the walk, since the objectSelector still matches an
UPDATE whose old object lacks the label. Pods go through it regardless,
as each Pod admission also sets the scheduling class.
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
## What this PR does
This replaces the open Renovate PRs with one rollup. Go modules move to
their newest releases that keep the Kubernetes libraries and
controller-runtime where they are, the toolchain and builder images move
to Go 1.27.1, and the first-party code gets the Go 1.27 modernizers.
Tests for the code the bumps touch come first, in their own commits, and
pass on both the old and the new dependencies.
Before the bumps:
- The e2e sandbox image builds again: dl.min.io answers 410 for mc since
MinIO archived it, so mc now comes from its last GitHub release with a
pinned checksum.
- Renovate stops offering tags of the apimachinery fork. They mirror
upstream, so #2999 is a downgrade that drops our patch.
- `make unit-tests` now runs the tests of the nested Go modules
(kubevirt-csi-driver, token-proxy, kubeovn-webhook, api/apps/v1alpha1).
They never ran in CI before.
- New tests: JWT verification in the dashboard token proxy, the kube-ovn
plunger metrics, cert-manager object idempotency in the tenant gateway,
values-file layering in the operator, and the kubevirt CSI driver
wrapper.
- One real fix. The lineage webhook decoded admitted objects with
encoding/json, so an int64 above 2^53 came back rounded. jsonpatch v2.5
compares numbers exactly and would emit a replace that rewrites such a
field in the pod. The webhook now decodes with apimachinery's json.
Held back on purpose, with the reasons in the commit messages:
- Kubernetes libraries stay on v0.35. #3001 needs the apimachinery fork
rebased onto release-1.37 first.
- cel-go v0.29.2 and etcd v3.6.15, the lines k8s.io/apiserver v0.35 is
built and tested with.
- grpc v1.83.2. v1.84.0 is affected by GO-2026-6443 and no v1.84 patch
release carries the fix yet.
- kubevirt.io/api and the CDI API stay on what upstream csi-driver
builds against. From CDI v1.60 it no longer compiles.
- api/apps/v1alpha1 keeps go 1.26.4, so consumers of the published types
are not forced onto Go 1.27.
- go fix runs without embedlit (Go 1.27-only syntax, breaks backports to
release branches on Go 1.26) and omitzero (would make metadata, spec and
status required in the published OpenAPI schemas), and one errorsastype
rewrite that did not compile is left out.
govulncheck reports nothing reachable in any of the five modules,
standard library included.
Supersedes #4311, #4310, #4309, #3452, #3993, #3927, #3902, #3805,
#3451, #3415, #3414, #3413, #3302, #4050 and #3363. #3001 stays open.
Not checked locally: the images that build upstream projects (kube-ovn,
kamaji, cilium, linstor-csi and others) with the Go 1.27 builder. The
image build jobs of this PR cover them.
Closes#4487.
### Screenshots
No UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
chore(deps): move the toolchain and builder images to Go 1.27.1 and raise Go dependencies to their latest compatible releases, closing the published advisories; the lineage webhook no longer rewrites int64 fields above 2^53 in admitted objects.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a restore resource policy option for Velero strategies.
* Added a test target to run Go tests across tracked modules.
* **Bug Fixes**
* Updated the end-to-end test environment to download and verify the
MinIO client from its current release location.
* Added coverage for token validation, webhook patching, controller
behavior, and reconciliation.
* **Maintenance**
* Updated the Go toolchain, dependencies, and container image versions.
* Refreshed pinned workflow actions and base image digests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
A definition declaring an application kind that another definition already
declares now fails when it is applied, instead of sitting next to the first
one where it has no effect. An update that keeps its kind is allowed, so a
duplicate that predates the check keeps upgrading, and so is a second
definition from the same Helm release, since a release that renames its
definition object creates the new one before it deletes the old one.
The webhook is registered with failurePolicy Ignore: ApplicationDefinitions
ship with the platform itself, so an outage of the webhook must never block
a bootstrap or an upgrade, and a failure to read the existing definitions
lets the request through with a warning rather than answering with an error,
which Ignore would not cover. It is the feedback layer; which definition
manages a kind is decided by the ownership rule whatever it lets through.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Nothing enforced that spec.application.kind is unique, and each consumer that
maps a kind back to a definition resolved a duplicate its own way: the
chartRef reconciler let every definition rewrite every release of its kind,
the lineage webhook and the CA projection reconciler took whichever came
first in an unordered list, and cozystack-api registered one resource per
definition. A second definition declaring a kind already in use could
therefore repoint that kind's tenant releases at its own artifact.
Add one ownership rule, the definition created first owns the kind with ties
broken by name, and route every consumer through it. A definition that
arrives later with a kind already in use no longer manages that kind's
releases whatever it is named: the chartRef reconciler skips it with a
Warning event and re-reconciles the next owner when the current one goes,
cozystack-api registers the owner only and logs the rest, its restart hash
moves when ownership moves, and the lineage and CA projection lookups
resolve the owner.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.
Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
With no decoder injected, the webhook decodes the admitted object with
encoding/json, which turns every number into float64, so an int64
above 2^53 comes back rounded. jsonpatch up to v2.4 compared both
sides as float64 and hid this; from v2.5 it compares json.Number, and
the webhook would emit a replace that silently rewrites such a field.
Decode with apimachinery's json, which keeps integers as int64, and
pin that the patch touches only the lineage labels and scheduling.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Import SchedulingClassAnnotation, SchedulingClassLabel, SchedulerName,
and GVR components from the cozystack-scheduler/pkg/apis sub-module
instead of defining them locally. This ensures the webhook and scheduler
stay in sync on label/annotation keys.
Also standardize the namespace label from scheduling.cozystack.io/class
to scheduler.cozystack.io/scheduling-class for consistency with the
scheduler, and resolve scheduling class from the owner Application CR
(via a new SchedulingClass() stub method) before falling back to the
namespace label.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Before injecting schedulerName and annotation, the webhook now
checks that the referenced SchedulingClass CR actually exists.
If the CRD is not installed or the CR is missing, injection is
skipped so that pods are not stuck Pending on a non-existent
scheduler.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
When a namespace carries the scheduling.cozystack.io/class label,
the webhook injects schedulerName=cozystack-scheduler and the
scheduler.cozystack.io/scheduling-class annotation into every Pod.
This covers all workloads in the tenant namespace regardless of
whether the operator CRD supports schedulerName natively.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Rename the CRD and all related types for better clarity:
- CozystackResourceDefinition -> ApplicationDefinition
- CozystackResourceDefinitionList -> ApplicationDefinitionList
- CozystackResourceDefinitionSpec -> ApplicationDefinitionSpec
- All related nested types updated accordingly
Updated components:
- API types and generated deepcopy code
- Controllers and reconcilers
- Dashboard, lineagecontrollerwebhook, crdmem packages
- CRD YAML definition and Helm chart
- All 25 cozyrds YAML manifests
- Migration scripts and documentation
Added migration 23 to remove old cozystack-resource-definition-crd HelmRelease.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Add three application metadata labels to HelmRelease:
- apps.cozystack.io/application.kind
- apps.cozystack.io/application.group
- apps.cozystack.io/application.name
- Replace shouldIncludeHelmRelease filtering with label-based filtering
in Get, List, and Update methods
- Always add kind and group label requirements in List for precise filtering
- Update CozystackResourceDefinitionController to watch only HelmReleases
with cozystack.io/ui=true label
- Update LineageControllerWebhook to extract metadata directly from
HelmRelease labels instead of using mapping configuration
- Add functionality to update HelmRelease chart from CozystackResourceDefinition
using label selectors
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Some HelmReleases use `chartRef` instead of `chart`. If the
lineage webhook finds such a HelmRelease, a nil pointer dereference
happens. This patch adds a nil check to guard against this.
```release-note
[lineage] Add a nil check to guard against HelmReleases with a nil
.spec.chart field when traversing the ownership tree.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The lineage-controller-webhook makes a lot of outgoing API calls for
every event it handles, contributing to a high API server latency,
increasing the number of in-flight requests and generally degrading
performance. This patch remedies this by separating the lineage
component from the cozystack-controller and deploying it as a separate
component on all control-plane nodes. Additionally, a new internal label
is introduced to track if a resource has already been handled by the
webhook. This label is used to exclude such resources from
consideration. Addresses #1513.
```release-note
[lineage] Break webhook out into a separate daemonset. Reduce
unnecessary webhook calls by marking handled resources and excluding
them from consideration by the webhook's object selector.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Since the Cozystack extension API can now change dynamically while there
are live clients (the lineage webhook) querying this API, the REST
mapper of the client should "expect" that things may change and refresh
their discovery information when they get a cache miss to see if new
kinds have been registered.
```release-note
[lineage] Use an auto-refreshing RESTMapper in the webhook's API client
that tries to update its API discovery info when it fails to GET a
resource kind that was previously not registered in its schema.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch extends the resource-selecting function of the webhook to
also apply selectors to ingresses and services, like has been already
done for secrets. The Cozystack resource definitions have been upgraded
to contain two more fields: `ingresses` and `services` and populated
with counterparts of the legacy selectors from the dashboard roles.
```release-note
[controller, api] Enable marking ingresses and services as user-facing
and implement selectors for existing CozystackResourceDefinitions.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch carries the selectors for secrets to be shown to end users
over from the legacy dashboard-resourcemap roles into the new
CozystackResourceDefinition selectors. Also a {{ .namespace }} template
variable is added to the variables supported in the `resourceNames`
field in the selector.
```release-note
[controller,api] Support {{ .namespace }} in `resourceNames` resource
selectors, add whitelist of secrets to show to end-users.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch refactors the secret selectors to use the
`internal.cozystack.io/tenantresource` label for managing secret
visibility and removes any selectors based on it or the previous
`apps.cozystack.io/tenantresource` label, the idea being that this label
will only ever be set by the controller.
```
[controller,api] Refactor labels for the secret selector.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch implements name-based selectors for
`CozystackResourceDefinitions.spec.secrets`. Application developers may
now specify secrets that should or should not be visible to end users by
specifying a `resourceNames` field with a string slice of acceptable
names. This will, for instance, let developers exclude a secret like
`postgres-dbname-superuser` that has a predictable name even if it does
not have predictable labels. Simple templates are supported, so
`postgres-{{ .name }}-superuser` is also a valid entry under
`resourceNames`.
```release-note
[lineage, controller] Let application developers determine resource
visibility for end users by name, as well as by labels.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
A new dashboard based on https://github.com/PRO-Robotech/openapi-ui
project
<img width="1720" height="1373" alt="Screenshot 2025-08-01 at 09-01-00
OpenAPI UI"
src="https://github.com/user-attachments/assets/7ae04789-24ec-4e4b-830b-6f16e96513eb"
/>
<img width="1720" height="1373" alt="Screenshot 2025-08-01 at 09-01-14
OpenAPI UI"
src="https://github.com/user-attachments/assets/ca5aa85d-43f0-4b5b-b87a-3bc237834f10"
/>
<img width="1720" height="1373" alt="Screenshot 2025-08-01 at 09-02-05
OpenAPI UI"
src="https://github.com/user-attachments/assets/ebee7bfa-c3ac-4fe6-b5e1-43e9e7042c6a"
/>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[cozystack-api] Implement TenantNamespace, TenantModules, TenantSecret and TenantSecretsTable resources
[cozystack-controller] Introduce new dashboard-controller
[dashboard] Introduce new dashboard based on openapi-ui
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
[cozystack-controller] Introduce new dashboard-controller
[dashboard] Introduce new dashboard based on openapi-ui
Co-authored-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Many resources created as part of managed apps in cozystack (pods,
secrets, etc) do not carry predictable labels that unambiguously
indicate which app originally triggered their creation. Some resources
are managed by controllers and other custom resources and this
indirection can lead to loss of information. Other controllers sometimes
simply do not allow setting labels on controlled resources and the
latter do not inherit labels from the owner. This patch implements a
webhook that sidesteps this problem with a universal solution. On
creation of a pod/secret/PVC etc it walks through the owner references
until a HelmRelease is found that can be matched with a managed app
dynamically registered in the Cozystack API server. The pod is mutated
with labels identifying the managed app.
```release-note
[cozystack-controller] Add a mutating webhook to identify the Cozystack
managed app that ultimately owns low-level resources created in the
cluster and label these resources with a reference to said app.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Many resources created as part of managed apps in cozystack (pods,
secrets, etc) do not carry predictable labels that unambiguously
indicate which app originally triggered their creation. Some resources
are managed by controllers and other custom resources and this
indirection can lead to loss of information. Other controllers sometimes
simply do not allow setting labels on controlled resources and the
latter do not inherit labels from the owner. This patch implements a
webhook that sidesteps this problem with a universal solution. On
creation of a pod/secret/PVC etc it walks through the owner references
until a HelmRelease is found that can be matched with a managed app
dynamically registered in the Cozystack API server. The pod is mutated
with labels identifying the managed app.
```release-note
[cozystack-controller] Add a mutating webhook to identify the Cozystack
managed app that ultimately owns low-level resources created in the
cluster and label these resources with a reference to said app.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
mattia-eleuteri