Commit Graph
7777 Commits
Author SHA1 Message Date
Andrey KolkovandClaude Opus 4.8 5bcdb8cfb8 fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.

1. Archive-vs-recovery serverName collision. The driver set the restored
   cluster's WAL-archive serverName equal to the recovery source
   (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
   chart pinned the archiver serverName to the release name. In place those
   coincide, so the cluster archived onto the source's own WAL prefix and
   barman-cloud-check-wal-archive refused to start ("Expected empty
   archive"). Fixed by giving the restored cluster a fresh
   bootstrap.newServerName, keyed by the RestoreJob UID (unique per
   restore, stable across reconciles); the chart archives there while
   externalClusters still recovers from bootstrap.serverName. This is what
   the manual CNPG strategies hit (serverName defaults to the cluster
   name); the cozy-default flow dodged it (serverName is <ns>-<app>).

2. useSystemBucket guard. The restore populates explicit S3 coordinates
   (destinationPath / s3CredentialsSecret / endpointCA) but left
   backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
   incompatible with useSystemBucket" guard failed the HelmRelease upgrade
   and the cluster never re-rendered. This is what the cozy-default
   (system-bucket) flow hits on an in-place restore. Fixed by clearing
   useSystemBucket in the restore patch, since the app is now on the
   explicit-creds path. The driver's app view (postgresapp.Backup) gains
   the field, without omitempty so a false survives the merge patch.

Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-25 17:31:53 +04:00
Kirill Ilin 517e33f67b fix(keycloak): pin the Vault leg to the configured CA and roll on rotation
SSL_CERT_FILE alone does not narrow trust. crypto/x509 replaces only its file
list and then scans SSL_CERT_DIR unconditionally, and the pinned proxy image
ships /etc/ssl/certs/ca-certificates.crt, so the connection guarding the KEK
trusted the private CA plus every public root — a mis-issued certificate for the
Vault hostname was still accepted. Set SSL_CERT_DIR to the mount as well, which
collapses the pool to the CA the operator configured.

An unvalidated caBundle failed open on top of that: a non-PEM value is dropped
silently by AppendCertsFromPEM, leaving the proxy on the image's roots with no
error at all. Reject it at render time, the way backend and auth already are.

Rotating an inline caBundle had no effect either. The pod template did not
change, so no new pods, and crypto/x509 memoizes the pool per process — the
upgrade reported success while the old CA stayed in use until some later
restart. Derive a checksum/vault-ca annotation from the bundle so the rollout
happens with the change.

Also add vault.caSecretKey: the projected key was hardcoded to ca.crt, and a
bundle stored under another name (a cert-manager Certificate from an ACME issuer
emits no ca.crt at all) wedged the pod at mount time with no way to correct it
from values.

Drop the claim that the database leg verifies against its own CA pool: the chart
never sets KKP_BACKEND_CA_FILE, so that leg is plaintext today, as the note 30
lines above already says. The conclusion holds — it never consults the system
trust store — but for the other reason.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-08-25 18:11:40 +05:00
Kirill Ilin 98ef5413f4 fix(keycloak): keep the Vault CA options inert on the static KMS backend
The volume and mount that consume the Vault CA sit at pod level, outside the
vault-transit env block, while $vaultCASecret was seeded from caSecretName
regardless of the backend. With kms.backend=static and caSecretName set, the
Deployment mounted a Secret the chart never renders, so the proxy — the single
choke point for all Keycloak DB traffic — would sit in ContainerCreating.

Gate both CA keys, and the mutual-exclusion check between them, on
vault-transit, so the whole vault.* subtree behaves the same way under static:
ignored. Project only ca.crt from a pre-seeded Secret, which keeps a
cert-manager tls.key out of the container and turns a bundle stored under
another key into a mount failure rather than an opaque TLS error later.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-08-25 18:11:40 +05:00
Kirill Ilin 3531553176 feat(keycloak): let the KMS proxy trust a private Vault CA
The proxy could only reach a Vault whose certificate chains to a publicly
trusted root, so a Vault fronted by an internal load balancer with a
self-signed certificate was unreachable over HTTPS.

Allow the CA to be supplied inline (caBundle, the chart renders the Secret) or
as a pre-seeded Secret (caSecretName), mounted and exposed as SSL_CERT_FILE.
That redirects only the system trust store, which nothing but the Vault client
consults — the database leg verifies against its own CA pool.

Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-08-25 18:11:39 +05:00
Andrei Kvapil 652231e0cc docs: add public ROADMAP.md (2026-2028) (#2666)
## Summary

This PR adds a public `ROADMAP.md` covering the project's direction over
the two-year window May 2026 – May 2028, aligned with the CNCF Sandbox →
Incubating → Graduated trajectory.

## What

A single living document organized into 11 strategic tracks:

1. Platform Core
2. Testing & Conformance
3. Security, Certifications & Compliance (largest track — covers OSPS
Baseline L1–L3, OpenSSF Best Practices Passing/Silver/Gold, Scorecard,
SLSA Build L1–L3, CLOMonitor, CNCF TAG Security self-assessment and
third-party audit, EU CRA posture, SOC 2 evidence pack, PCI DSS v4.0.1
reference architecture, ISO/IEC 5230)
4. Storage Independence (blockstor)
5. Host OS Strategy
6. AI/ML Platform
7. AI Inside Cozystack
8. Marketplace & Application Ecosystem
9. Multi-Cluster & Federation
10. Developer Experience & Reference Architectures
11. Ecosystem Standardization & Community

## Why

The CNCF Incubation review (targeted for late Q3 2026) expects a public,
readable roadmap in-tree. The existing Project V2 board
([#1](https://github.com/orgs/cozystack/projects/1)) is the issue-level
source of truth, but a multi-quarter narrative view is needed for
adopters, downstream vendors, and TAG reviewers.

## Preview

The document is a living artifact: updated quarterly via a public
roadmap review, with material changes routed through CzEP (Cozystack
Enhancement Proposal) in \`cozystack/community/design-proposals/\`.

## Notes for reviewers

- Content is intended as a discussion starting point; expect refinements
in subsequent PRs once SIGs form.
- Certification timelines are aspirational targets, not commitments —
refinement expected once a SIG-Security charter is ratified.
- Cross-references to other Cozystack repositories (blockstor,
community, cozystack-ui) reflect current state on \`main\` as of
2026-05-18.

## Test plan

- [ ] Markdown renders correctly on GitHub
- [ ] All internal repo links resolve (\`./GOVERNANCE.md\`,
\`./MAINTAINERS.md\`, \`./SECURITY.md\`, \`./CONTRIBUTOR_LADDER.md\`)
- [ ] All external references in §10 resolve
- [ ] Maintainer review of certification timelines (Track 3) for
feasibility
- [ ] Maintainer review of SIG formation timeline (Track 11)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated the public roadmap for August 2026 with clearer commitment and
direction horizons.
* Refreshed current-state information, release details, in-flight
initiatives, and milestone planning.
* Added platform milestones for service exposure, readiness, quotas,
autoscaling, and secrets management.
* Strengthened end-to-end quality gates and reproducible benchmarking
guidance.
* Revised plans for AI/ML, self-hosted assistance, CLI development,
governance, conformance, risks, and maintenance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-25 14:24:53 +02:00
IvanHunters 32ec1b5a00 fix(migrations): correct unadopted-record count semantics and stale comments
- The unadopted-record WARNING said "N pool(s)" but counted SKIPPED_FILE lines,
  and the foreign-owner branch writes one line per object (so a single pool with
  MD+MHC+WM reported "3 pools"). Reword to "N record(s) ... one line per object
  or pool ... not a distinct-pool count" so the number does not mislead.
- The inline-chart restamp comment implied the cozystack-apps source is enforced;
  it is not (matching by chart name only is a deliberate trade-off: gating on the
  historical sourceRef risks under-matching a real legacy parent, which loses its
  workers, whereas the loose match's worst case is a harmless mislabel). Spell out
  the trade-off and that the render guard is the data-loss backstop.
- The overflow comment cited "up to 42 chars per rest.go"; the enforced cap is 32
  (maxKubernetesClusterName = 53 - 17 - 4). Correct it and note legacy/hand-created
  names can still be longer, which is why the overflow branch exists.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 14:40:20 +03:00
IvanHunters 3630842172 fix(computeplane): reject non-RFC-1123 pool keys at render time
The pool key becomes the pool HelmRelease name and its CAPI object names, which
must be RFC-1123 labels, but only the length was guarded. An uppercase/underscore
key rendered an invalid metadata.name the apiserver rejects, and a key containing
':' rendered invalid YAML that failed the whole module render -- a non-uniform,
late failure next to the guarded length case. Add a render-time RFC-1123 guard on
the key alongside the length guard (mirrors migration 56's child-name guard), and
pin the length guard's derived remedy budget (15) in the overflow test so a wrong
computation is caught, not just the presence of the error.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 14:40:20 +03:00
Timur TukaevandAndrei Kvapil 2619c91f09 docs: add public ROADMAP.md (2026-2028)
Adds an in-tree public roadmap giving adopters, downstream vendors and
CNCF TAG reviewers a multi-quarter narrative view alongside the
issue-level Project V2 board.

The document covers twelve tracks, the certification and compliance
programme, the standardization strategy and the SIG formation plan. It
marks which horizons are commitments with owners and capacity behind
them and which are direction, and it states plainly that the board
remains the issue-level source of truth.

Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
Assisted-By: Claude
Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-08-25 13:21:57 +02:00
IvanHunters c0fcdd767c fix(migrations): discover pre-chartRef parents via inline spec.chart in the restamp
The restamp discovered unlabelled parent kubernetes releases by spec.chartRef
only, but that misses the population it exists for. The apps.cozystack.io/
application.kind label was introduced 2025-12-19 (fdca49838); spec.chartRef only
2026-01-14 (43da779ee). So any parent new enough to carry chartRef already
carries the label and is handled by the label-selected sweep, while every
genuinely unlabelled parent predates chartRef and instead carries an inline
spec.chart (chart name "kubernetes" from the cozystack-apps HelmRepository).
The chartRef-only selector therefore matched none of the real targets. Match both
shapes. This is a seamless-adoption improvement, not a data-loss fix: the chart's
own render-time guard already fail-closes (blocks the upgrade rather than pruning)
for any parent the restamp misses. Exact chart-name match plus the kubernetes-
name-prefix guard keep it from touching other apps or kubernetes-nodes children;
covered by an inline-shape restamp test and a non-kubernetes decoy test.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 13:50:14 +03:00
IvanHunters c6b6755eec fix(migrations): record foreign-owner skips, harden record_skip, scope restamp to kubernetes- names
Three hardening fixes on migration 56's new code:
- The foreign-owner branch in adopt_one pins the object prune-proof but does not
  adopt it (the child release can then stay Failed on the ownership conflict),
  yet it was the one warn+pin+skip path that did not record itself in the durable
  unadopted ConfigMap -- so an operator relying on that record to find stuck
  pools would miss it. Record it like every sibling branch.
- record_skip ran a bare append under set -e inside skip branches that are
  designed never to abort; a failed write (e.g. /tmp full) could deadlock the
  fleet upgrade for one tenant. Make it fail-open, matching the branches' intent.
- The restamp selector matched any release by chartRef; scope it to names
  starting with kubernetes- so it cannot mislabel (and then have the sweep skip)
  a non-standard release, keeping restamp and the sweep's contract in sync.
Also fix the previously-vacuous negative restamp test (its fixture lacked the
chartRef, so it never reached the label guard it claimed to check).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 13:28:58 +03:00
IvanHunters c2ee58756e fix(computeplane): allowlist pool values instead of denylisting, derive length-guard prefix
The pool-value splice used omit (a denylist), which leaks every key it does not
name. Since this module runs untrusted tenant code and the NodeGroup schema has
no additionalProperties:false, a tenant could pass the platform-injected
_cluster object (cluster-domain/oidc/dns01/proxy, delivered via cozystack-values
and merged UNDER spec.values, so a tenant key wins) or nameOverride straight
through to the pool chart -- and any future kubernetes-nodes top-level value
would silently re-open the same hole. Switch to pick: pass through only the
enumerated per-pool shape fields, so everything cluster-level or module-owned is
dropped by construction (fail-closed for new fields). Also derive the
length-guard's release-name prefix from $clusterRelease so it cannot drift from
the name actually rendered. Regression-tested: _cluster and nameOverride no
longer reach the child values; maxUnhealthy and the module wiring still do.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 13:28:58 +03:00
Alexey Artamonov 30af0e1f60 docs(postgres-operator): update values comment for the native 1.30.0 bump
The explanatory comment in values.yaml still described the removed
patches/cloudnative-pg-1.28.2.patch mechanism and the old 0.27.1/1.28.1
versions, and pointed at an "edit the patch" workflow that no longer exists.
Rewrite it to describe the native 0.29.0/1.30.0 chart bump (operator and CRDs
aligned by construction, no patch step) while preserving the still-valid
guidance: do not pin image.tag, keep the operator and CRDs in lockstep, since a
mismatched pin prunes CRD status fields and fails backups cluster-wide.

Addresses review feedback on #3951 (IvanHunters, CodeRabbit).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-25 12:58:25 +03:00
IvanHunters 3d1a83f419 fix(migrations): make the unadopted-pools record publish best-effort
The durable-record ConfigMap write sat under set -euo pipefail right before the
version stamp, so a permanent failure (e.g. the record exceeding etcd's object
size limit at a very large skip count) would abort the migration and deadlock
the fleet upgrade with no path out -- even though every skipped pool was already
pinned prune-proof earlier in the sweep, so the workers are safe regardless of
whether the record lands. The record is a convenience artifact, not a safety
mechanism, so swallow its failure with a warning instead of failing closed.
Covered by a bats case that fails the publish and asserts the run still stamps.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 12:38:28 +03:00
IvanHunters 12fc8365bb fix(kubernetes): make the worker-adoption guard remedy actionable
The guard's refusal message told the operator to run the platform migration,
but the case that reaches it is precisely one the migration already ran and
stamped for (its slot was renumbered out on a backport, or the parent
HelmRelease predates the application.kind label and the selector skipped it) --
re-running does nothing. Name the manual recovery instead: annotate
resource-policy=keep on the listed MachineDeployments, then stamp the
cozystack-version ConfigMap back below the worker-adoption migration so it
re-runs. Also note in the Phase 2 README that computeplane clusters render
their worker pools as separate KubernetesNodes releases too.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 12:28:47 +03:00
IvanHunters 952901776a fix(migrations): adopt unlabelled parent kubernetes releases and record unadopted pools in migration 56
The worker-adoption sweep selected parents by the apps.cozystack.io/
application.kind label, which postdates v1.2. A parent kubernetes HelmRelease
older than that, or one restored by hand, lacks the label: the sweep skipped
it, stamped, and the parent's control-plane-only re-render then pruned its
un-pinned workers with no pin and no warning. Discover such releases by their
chartRef instead and restamp the three application labels before the sweep
(additive, cannot corrupt a release, and repairs it for cozystack-api too).

Every warn+pin+skip branch leaves a pool pinned prune-proof but unadopted,
recorded only in the Job log, which the next upgrade reaps; the migration does
not re-run, so an operator cannot find those pools afterward. Accumulate the
skips and publish them as configmap/cozystack-migration-56-unadopted before
the stamp.

Extend the fake kubectl and add bats coverage for the restamp (positive and
negative) and the unadopted-ConfigMap paths.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 12:28:46 +03:00
IvanHunters 34e801cb9e fix(computeplane): stop tenant pool values leaking cluster-level keys and overflowing release names
The per-pool child HelmRelease spliced the tenant nodeGroup dict verbatim
(toYaml) after setting cluster/clusterReleaseName, and the NodeGroup schema
has no additionalProperties:false. A tenant could therefore (a) set a
duplicate cluster/clusterReleaseName key that wins the YAML merge and
re-parents the pool onto a tenant-controlled cluster, or (b) pass talos/
version/images through a shape-only knob and pick the worker OS image,
installer and kubelet version. omit those five keys at the splice so only the
pool's own shape flows through (tolerated extras like maxUnhealthy still do).

Separately, each pool renders a 38-char fixed release-name prefix against
Helm's 53-char cap, so a nodeGroup key over 15 chars created a HelmRelease the
apiserver accepts but Flux never reconciles, with no other signal. Guard the
length at render time, next to the ingress-role guard, mirroring the migration
overflow branch and the aggregated API's validateNameLength.

Also refresh the generated values doc: the module now materialises its own
default md0 and pool storageClass falls back to the kubernetes-nodes chart
default, not the wrapped kubernetes app.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-25 12:28:32 +03:00
Andrei Kvapil 6663c44dd5 ci(labeler): map the kubeovn-webhook scope to area/networking
The package has its own scope in commit history but no entry, so its
PRs land on the area/uncategorized fallback. It is a kube-ovn admission
component, which is the same area as the kube-ovn scope beside it.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-24 23:55:36 +02:00
Andrei Kvapil cdc34d6e34 fix(kubeovn-webhook): restrict the webhook port to the API server
The /mutate-pods handler returns namespace-derived annotations to any
caller that reaches it (GHSA-g883-q79m-8225), and until now nothing
stopped an arbitrary in-cluster pod from being that caller.

Client-certificate verification, added in #2698, cannot close this on
its own. Enabling --client-ca-file requires the API server to *present*
a certificate, which it does only when its AdmissionConfiguration names
a kubeConfigFile for the webhook plugins. Checked on a running
Cozystack control plane: the AdmissionConfiguration carries a
PodSecurity block and nothing else, so no certificate is offered at
all. Flipping to RequireAndVerifyClientCert there would reject the API
server itself, and with failurePolicy: Fail that stops pod creation
cluster-wide. No amount of log observation changes that — it is a
property of the control plane's configuration, not of the traffic.

So the network is where the caller can actually be constrained. The
policy denies pod and world traffic to the serving port and needs no
carve-out: the API server runs on the host network, so it is not a
Cilium endpoint and fromEndpoints never matches it. Unlike the
ingress-nginx admission webhook this component exists only on the
management cluster, so there is no konnectivity-tunnelled caller to
exempt either, and no forgeable pod label anywhere in the selector.

It is a CiliumNetworkPolicy rather than a core NetworkPolicy, and that
distinction is why it works at all: kube-ovn runs with ENABLE_NP: false
and programs no ACLs, so a core NetworkPolicy would be created and
never enforced. The values comment claiming NetworkPolicy was simply
not an option is corrected — it was true only of the core kind.

Deny-only with default-deny disabled, matching the ingress-nginx
admission policy: putting these pods behind a default-deny would mean
enumerating every port they serve and breaking traffic on the first one
forgotten.

Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-24 23:54:33 +02:00
Andrei Kvapil 4689b2b844 feat(kubeovn-webhook): add client-certificate verification, off by default (#2698)
## What this PR does

Adds client-certificate verification to the kube-ovn webhook, and turns
it on only when an operator supplies a CA bundle.

`--client-ca-file` is new and empty by default. With it empty the server
sets `tls.RequestClientCert`: it asks callers for a certificate, records
the issuer of whatever arrives, and serves the request either way. Set
the flag and the server switches to `tls.RequireAndVerifyClientCert`, so
a caller without a certificate signed by that CA is refused at the TLS
handshake.

**This is step one of two, and on its own it does not close
[GHSA-g883-q79m-8225](https://github.com/cozystack/cozystack/security/advisories/GHSA-g883-q79m-8225).**
With the default empty value nothing is rejected — the webhook still
answers an unauthenticated caller, and the only change is that the
caller's issuer now appears in the logs. That is deliberate: switching
straight to `RequireAndVerifyClientCert` would break every cluster whose
API server sends no client certificate, and there is no way to know from
here which those are. The logs are how we find out.

Step two, in a follow-up: read a day of issuer logs on a Talos cluster
and on `isp-full-generic`, confirm the API server presents a certificate
we can pin, then set the CA bundle so verification becomes mandatory.
The advisory should be published as fixed only after that lands — not
with this PR.

An earlier revision also carried a NetworkPolicy restricting ingress to
`kube-system`. It was dropped, and the title has been corrected to
match; only the TLS path is left here.

### Screenshots

N/A — no user-facing or UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the trigger map against the diff: no row matches. The change
touches one system package and the Go image it builds — no app package,
no `values.schema.json`, no `ApplicationDefinition`, no platform
variant, no node prerequisite.

### Release note

```release-note
feat(kubeovn-webhook): the webhook now accepts --client-ca-file. Left empty (the default) it requests and logs client certificates without requiring them; set to a CA bundle it rejects callers that do not present a certificate signed by that CA.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional client-certificate verification for the pod mutation
webhook through configurable CA settings.
* Added separate rate limits for requests with and without client
certificates.
* Added remote-address logging and clearer verification status
reporting.
* **Security**
* Requests continue to be forwarded while certificate observations are
recorded when verification is disabled.
* Serving-certificate updates remain supported; client CA changes
require a webhook restart.
* **Documentation**
* Documented client CA configuration and current certificate rotation
limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-24 23:35:31 +02:00
Andrei Kvapil 35668dd27c feat(dashboard): rework the Marketplace entry point and the Admin portal (#3928)
## What this PR does

Makes the Console the dashboard's entry point, reworks the Marketplace
into an ordinary destination, and turns the Admin portal into the place
where tenant-wide administration actually lives.

**The Console is where the dashboard opens.** Visiting the dashboard
without a path lands on the Console, which shows what the tenant already
runs — what someone opening it almost always came for — and the header
gives the Console the primary emphasis. The Marketplace is a place you
go deliberately, when the intent is to add something new, and it is one
click away from its own tab, from the Console overview and from the
command palette.

**Marketplace** opens the grouped application list directly. The "Choose
a product to deploy" landing page and its Show-all-apps button are gone,
and category links keep filtering to a single category.

**Administration moves from Console to Admin.** Tenants, Modules,
External IPs and the per-tenant Info page need no special permission, so
the Admin tab is now always visible instead of being gated. The two
cluster-wide operator areas keep their own independent gates, so
Capacity and Backup Classes still appear only for users who can open
them. Resource routes became portal-aware, so a detail, edit or order
page opened from Admin stays under `/admin` instead of jumping back to
`/console`.

**Tenants** is a hierarchy tree instead of a flat table. It is built
from `TenantNamespace` labels (`tenant.cozystack.io/<ancestor-ns>`), so
a tenant whose parent is not accessible attaches to its nearest visible
ancestor rather than disappearing, and rows are named relative to that
parent — a sub-tenant reads as `crpjxhwm`, not `whmcs-crpjxhwm`. Each
row carries its own actions: open that tenant's Info, or create a
sub-tenant at any level of the tree.

**Modules** is rebuilt around the same tree. It previously showed only
the selected tenant's modules as Enabled or Disabled cards, which meant
a module provided by a parent tenant looked disabled, and clicking it
navigated to a Tenant CR that does not exist in the child's namespace —
an endless spinner. The page now covers every tenant the user can
access, with each tenant's modules as chips showing the module icon, the
namespace running it and its readiness. Availability comes from the
`namespace.cozystack.io/<module>` label on each `TenantNamespace`, which
names the providing namespace: the tenant itself for a local module, an
ancestor for an inherited one. An inherited chip is shown only when that
ancestor is out of the user's reach, since otherwise its own chip
already shows the module. Clicking any chip opens the module in
whichever tenant runs it. Tenants whose whole subtree has no modules
collapse into a "+ N tenants" branch.

**External IPs** lists LoadBalancer services across every accessible
tenant in one table with a Namespace column, instead of only the
selected tenant. A namespace whose services the user cannot list
contributes no rows rather than failing the page.

Two navigation fixes come along with this. Detail pages offered a fixed
tab set, so Info showed Workloads, Services and Ingresses tabs it never
fills — tabs now appear only for resource groups the instance actually
owns, probed through the tabs' own list refs so they share one cache
entry and one watch. Back and the sidebar followed the resource plural
rather than the entry point, so a module page sent the user to a
one-item list and highlighted nothing; module pages now return to
Modules and keep it highlighted, while Info returns to Tenants.

Finally, copying a secret key no longer requires revealing it on screen
first, and the button briefly turns into a check mark so it is visible
that the value reached the clipboard.

### Screenshots

<img width="1720" height="1374" alt="Screenshot 2026-08-20 at 21-08-30
Cozystack"
src="https://github.com/user-attachments/assets/8499577c-8f9e-4df9-a6bb-df507497a4a0"
/>
<img width="1720" height="1374" alt="Screenshot 2026-08-20 at 21-08-40
Cozystack"
src="https://github.com/user-attachments/assets/01d256e9-2585-41fd-8070-b910f03d8f3f"
/>

### Downstream repositories

The diff touches only `packages/system/dashboard/images/console`. The
console is vendored in this repository (`cozystack-ui` is archived), and
the change adds no package, no values schema, no `ApplicationDefinition`
semantics and nothing under `hack/`, so no repository in the trigger map
is reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(dashboard): the dashboard now opens on the Console rather than the Marketplace, the Marketplace opens the application list directly, and tenant administration moves into the Admin portal, which is now always available. Tenants and Modules are shown as the tenant hierarchy across every tenant the user can access: a module provided by a parent tenant is no longer reported as disabled, and opening it lands on the tenant that actually runs it instead of hanging. External IPs covers every accessible tenant in one table, detail pages only offer tabs the instance has resources for, and a secret key can be copied while it stays masked.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Administration now includes dedicated tenant, module, external IP, and
information pages.
- Tenants appear in a collapsible hierarchy with creation, editing, and
detail navigation.
- Modules and external IPs are shown across accessible tenant
namespaces, including inherited modules.
  - Application details show tabs only for available resources.
  - Secret values can be copied while remaining masked.
- Console is now the primary landing page, with Marketplace available
separately.

- **Bug Fixes**
  - Improved sidebar highlighting for nested pages.
- Administration is accessible without operator permissions, while
restricted areas remain protected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-24 23:32:55 +02:00
Alexey Artamonov 3c78c0be56 feat(clickhouse): add configurable version parameter
Add a `version` value (v25.8, v25.3, v24.9; default v24.9) to the
ClickHouse chart, backed by files/versions.yaml and resolved by
templates/_versions.tpl for both the ClickHouse server and Keeper
images, so the Cozystack API returns the engine version (spec.version)
like the other managed-DB charts. Default v24.9 -> 24.9.2.42 keeps
existing installations byte-for-byte identical.

The maintainer-run hack/update-versions.sh generator intersects the
server/keeper Docker Hub tags byte-collated (LC_ALL=C) so comm keeps
tags across the X.9 -> X.10 boundary; freezes the default major to the
tag it already ships (read from versions.yaml, so the freeze follows the
default); errors on a dropped default or an unresolvable major; writes
atomically; and runs on stock macOS bash 3.2 / BSD. Covered by
hack/clickhouse-update-versions_test.bats (10 cases) and the
tests/version_test.yaml helm-unittest suite.

Refs: #1246
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-24 23:41:27 +03:00
Alexey Artamonov b20e8ee91a fix(postgres-operator): bump CloudNativePG to 1.30.0 for the Cluster scale-subresource selector
Bump the vendored cloudnative-pg chart 0.27.1 -> 0.29.0 (operator + CRDs
1.28.1-patched-to-1.28.2 -> 1.30.0) and drop the hand-maintained
cloudnative-pg-1.28.2.patch entirely — 0.29.0 is a published chart, so no
version-alignment patch is needed.

Why this is required
--------------------
Database read-replica autoscaling (community proposal
design-proposals/database-horizontal-autoscaling) drives the CNPG Cluster
scale subresource with a stock HPA (via KEDA). That only works when the
subresource serves status.selector, which cloudnative-pg#8996 added and
shipped in CNPG 1.28.4 / 1.29.2 / 1.30.0. cozystack shipped 1.28.2, whose
scale subresource has no selector, so the HPA fails with InvalidSelector.

Why 1.30.0 and not 1.28.4
-------------------------
The floor is >= 1.28.4, but the upstream chart repo publishes no chart for
1.28.4 (the 1.28 line stops at chart 0.27.1 / appVersion 1.28.1), so 1.28.4
could only be reached by hand-patching the 0.27.1 chart's operator image and
CRDs up to the backport. A live PoC on a dev cluster proved that path is
broken: the 1.28.4 operator running under the 0.27.1 (1.28.1) chart
templates crash-loops on a missing RBAC grant it needs but the older chart
does not provide —

  endpointslices.discovery.k8s.io is forbidden: ... cannot list resource
  "endpointslices"

— so status.selector is never populated and the HPA still cannot scale. The
hand-patch would have to port the operator's RBAC (and any other template
delta) from 1.28.1 to 1.28.4 by hand, with no published chart to diff
against. 1.30.0 (chart 0.29.0) is the only published chart that carries the
selector together with matching RBAC/CRDs/templates, and the 1.28 line is
EOL (2026-06-30), so 1.30.0 also keeps us on a supported release. cozystack
uses the barman-cloud plugin (not native barman, removed in 1.31) and no
CNPG Poolers, so the 1.30 deprecations and the known pooler bug do not
apply.

Update the version-guard unittest to 1.30.0 and assert the clusters CRD
scale subresource carries labelSelectorPath: .status.selector.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-24 22:46:57 +03:00
Andrei Kvapil 724354ffe1 fix(console): address the review findings on the Admin portal rework
Four of the five findings share one root cause: a node in the tenant
tree can attach to its nearest *visible* ancestor when its real parent
is inaccessible, and per-row actions then derived their target from
that bridged ancestor.

The Edit action on a bridged tenant row pointed at a Tenant CR that
does not exist. The CR lives in the real parent's namespace, so the
target now comes from realParentNamespace — and Edit is offered only
when that namespace is readable, because otherwise there is no CR the
user could open and the button could only ever dead-end. The test that
asserted two Edit buttons had frozen the broken target as expected
behaviour; it now names the rows it expects, and the Edit button gained
a title so that assertion means something.

An inherited module chip is rendered only when the providing tenant is
NOT visible, yet it was a live button: clicking it denied the instance
GET and switched the active tenant to a namespace outside the visible
set, which the selector then bounced to an arbitrary fallback. It is
now an inert badge that states where the module comes from.

InfoRedirect's missing-definition branch hardcoded /console, throwing an
admin out of the portal its own docstring promises to preserve.

The sidebar's alsoMatch used a bare startsWith, so /admin/apps would
light up for /admin/apps-v2. It now matches whole path segments.

And SecretsTab let a rejected clipboard write escape as an unhandled
rejection, leaving the button silent — a denied permission or an
insecure context was indistinguishable from a no-op. It now reports the
failure.

Declined: CodeRabbit asked to split `import { useK8sList, type K8sResource }`
into a separate `import type`. The inline modifier is valid under
verbatimModuleSyntax and is what 26 other imports in this console use,
so the change would make the file the odd one out.

Reported-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-24 10:45:20 +02:00
IvanHunters 5b08d48632 docs(migrations): pin DEFAULT_MD0 provenance to the pre-removal helper commit
The kubernetes.nodeGroups helper that DEFAULT_MD0 reproduces was deleted in this
line of work, so nothing in-tree diffs against it and a future edit to the
constant (instanceType/diskSize/...) would silently rename the content-hashed
KubevirtMachineTemplate and roll every implicit-md0 pool. Reference the
pre-removal definition (git show 92b9785da^:.../kubernetes/templates/_helpers.tpl)
so the canonical default stays diffable.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-24 11:26:44 +03:00
IvanHunters 8cf3af7309 test(kubernetes-api): cover the removed-fields warning at its Create/Update call sites
TestWarnRemovedKubernetesFields pinned the helper in isolation, so deleting
either call site (rest.go Create/Update) left it green while an operator editing
a Phase 2 removed field would silently get no warning. These drive Create and
Update end to end and assert the warning surfaces; removing a call site now
fails loudly (verified by mutation).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-24 11:24:56 +03:00
IvanHunters d93e136e23 fix(computeplane): provision worker pools via kubernetes-nodes after Phase 2 split
Phase 2 moved worker nodeGroups out of apps/kubernetes into apps/kubernetes-nodes,
but the ComputePlane module still passed nodeGroups to the wrapped kubernetes
chart, which now ignores it: a fresh ComputePlane cluster came up with zero
workers (and the pinned ingress-nginx addon stuck Pending), while existing ones
were unreachable by migration 56 (it only adopts kubernetes-* releases, not
computeplane-cluster).

The module now emits one kubernetes-nodes child HelmRelease per nodeGroup,
shaped like a natively created pool, and materialises the pre-split default md0
(roles: [ingress-nginx]) when nodeGroups is empty. The pool chart's clusterName
gains an internal clusterReleaseName override so pools attach to the module's
fixed computeplane-cluster release, which is off the aggregated API's
kubernetes-<cluster> convention; the default keeps every API-created pool
byte-identical.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-24 11:19:54 +03:00
myasnikovdaniil 334e2d8bf9 test(e2e): record the runner's SVM feature flags in the identity capture (#3945)
Adds one line to the runner identity record, taken from the `lscpu` read
it already does:

```
svm features: svm npt nrip_save vgif
```

`vgif` decides whether a guest that is itself running a hypervisor pays
a VM exit on every `CLGI`/`STGI`. Here that is the sandbox Talos nodes
hosting the tenant KubeVirt VMs, so it only bites at double nesting, and
the sandbox nodes' own boot is unaffected.

Across the archived reports no run missing the flag has passed a
tenant-Kubernetes suite: 0 of 62, against 11 of 12 with it. Exposure
varies between hosts of the same shape, so it is not a property of the
lane and cannot be recovered once the runner is recycled. Recording it
on every run, red and green, is what makes the correlation countable
rather than reconstructed by hand.

`apply` runs `hack/e2e-runner-identity.sh` before `docker run`, and
every workflow that runs the suite reaches it through `prepare-env`, so
this is captured once per run on both paths. `lscpu` is already
answering there: the `cpu model` line in today's records is parsed out
of the same table.

No new bounded read, no compiler on the runner, no sysfs. Gated on `svm`
so an Intel host's `vnmi` is not reported as an SVM feature.

Five tests for the reader. Identity suite 62 assertions, `hack/` sweep
732.

No behaviour change and no gating on the flag.
2026-08-23 19:51:45 +05:00
Myasnikov Daniil a94abe0971 test(e2e): record the runner's SVM feature flags in the identity capture
Adds one line to the runner identity record, taken from the lscpu read it
already does:

    svm features: svm npt nrip_save vgif

vgif decides whether a guest that is itself running a hypervisor pays a VM exit
on every CLGI/STGI. Here that is the sandbox Talos nodes hosting the tenant
KubeVirt VMs, so it only bites at double nesting, and the sandbox nodes' own
boot is unaffected.

Across the archived reports no run missing the flag has passed a
tenant-Kubernetes suite: 0 of 62, against 11 of 12 with it. Exposure varies
between hosts of the same shape, so it is not a property of the lane and cannot
be recovered once the runner is recycled. Recording it on every run, red and
green, is what makes the correlation countable rather than reconstructed by hand.

No new bounded read, no compiler on the runner, no sysfs. Gated on svm so an
Intel host's vnmi is not reported as an SVM feature.

Five tests for the reader. Identity suite 62 assertions, hack/ sweep 732.

No behaviour change and no gating on the flag.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-23 19:38:49 +05:00
Yan Bondarenko 296d3008d9 fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-08-21 15:51:53 +03:00
Andrei Kvapil 8554c036de fix(ingress-nginx): confine the konnectivity carve-out to kube-system
The admission-webhook deny exempted any endpoint carrying
k8s-app: konnectivity-agent, in any namespace. Because the rule
disables default-deny, falling out of the deny is the entire access
decision — so any pod anywhere could reach the webhook port by
labelling itself, which is precisely the access the policy exists to
remove.

k8s-app is an ordinary pod label a workload sets on itself. Nothing in
the platform constrains pod labels by admission, and Cilium keeps
k8s-app identity-relevant (its label filter ships no default
whitelist), so the forged label really does change the endpoint's
identity rather than being filtered out.

The namespace key is derived by Cilium from the pod's actual namespace
and cannot be set by the workload. Splitting the deny in two keeps the
carve-out and closes the hole: one selector for everything outside
kube-system with no label condition at all, one for kube-system
carrying the existing NotIn. Only the agent in its own namespace is
exempt.

Kamaji at the pinned 26.3.6-edge puts nothing but
k8s-app: konnectivity-agent on the agent pod template
(internal/resources/konnectivity/agent.go), so the label remains the
only available discriminator and the namespace confinement is what
makes it safe to key on.

Tests cover both halves: the outside-kube-system selector must carry no
label condition, and the kube-system selector must keep exactly the
namespace and label pair.

Reported-by: Daniil Myasnikov <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-21 14:41:27 +02:00
Aleksei Sviridkin 3c780acd41 test(e2e): stop blocking merges on the tenant node-join deadline (#3932)
## What this PR does

The tenant node-join wait in the two kubernetes e2e suites has stopped
discriminating. Nested virtualisation on the shared runners degrades far
enough that a worker which registers in minutes elsewhere can miss any
deadline this test can afford, so a red on that one wait says as much
about the machine the run landed on as about the product. What it
produces in practice is a re-run, and a signal that teaches people to
re-run is a signal that gets the meaningful reds re-run too.

The suite still fails on it. That is the part worth reading twice,
because it is what separates this from hiding a flake: the script exits
non-zero, Chainsaw records a failure, and its whole catch runs
(previous-instance container logs, the host snapshot, the data-plane
capture, the event dump), every one of which is keyed on the test
failing. The JUnit report says a test failed. What changes is one layer
up. The wait writes a `SOFT-RED-node-join.txt` marker beside its
diagnostics, and a new lane-side gate, `hack/e2e-node-join-soft-red.sh`,
reads that marker to decide whether the job blocks. Softening inside the
suite would have bought a green check by throwing all of that evidence
away.

The gate opens on one shape only, every failed test in the run carrying
that marker, and blocks on everything else, including every way it can
fail to answer the question: a report it cannot read, a failed test the
report does not name, a failure with no marker, a run whose report
records no failure at all. A marker is written only when `timeout`
reports 124 on that one wait, so an assertion after the join, or the
wait failing to run, stays an ordinary red. What the marker attributes
is the test rather than the individual failure: Chainsaw writes one
testcase per Test and combines that Test's step errors into its single
failure element, so a second step error rides soft alongside the
deadline, while a failed `catch`, `finally` or step-level `cleanup`
rides soft for a different reason worth keeping apart, since at the
pinned version those run with no step report and never reach the failure
element at all. The convention doc states both rather than working
around the granularity, since the suite stops at the wait and nothing it
exists to prove runs after the marker.

Only the lanes that block a merge ask the gate. `pull-requests` and
`e2e-fork` do; `nightly` and `e2e-tag` run the same suites and stay
hard. `e2e-tag` is the release candidate's own e2e check, where a soft
pass would ship a release whose tenant clusters brought up no worker
nodes. `nightly` is the only place an e2e failure reaches anyone who did
not open a pull request, since nothing in this tree sends a notification
anywhere, so the red job is the notification. A unit guard holds that
split in both directions and derives the lane set from the workflows
that run the suite, so a new lane has to be placed on one side or the
other before it passes.

One consequence of a softened red is worth naming rather than
discovering later: the job finishes green, so its result stops answering
whether the suite passed, and it has two kinds of consumer that each
break their own way. A step gated on the job failing silently stops
running. The SSH breakpoint in `pull-requests` is that case, and it is
the only interactive way into a sandbox whose node-join never finished,
so the E2E step records what the gate decided and the breakpoint's
condition reads it back; the step is label-gated, so the session opens
for a PR someone deliberately marked for debugging rather than on every
softened run. A job that reports the result silently starts reporting a
pass. That is the required `E2E Tests` commit status in both
pull-request lanes, and on a softened run it now says the node-join
deadline was missed, that the suite failed, and that the merge is not
blocked on it, instead of saying the suite passed. The handover is four
links long, and a guard reads each of them inside the block it has to
live in rather than anywhere in the lane: the step carrying the gate's
id writes the decision to the output sink, the job that runs the suite
publishes it naming that step, the job that reports the result reads it
off that job, and it branches on the value ahead of the branch that
would otherwise call the run a pass. A link matched file-wide passes
while it sits on a neighbour, and the value is then empty for good with
nothing to say so. The first link is read tighter still, and in both
directions: the write has to be an unconditional statement in the same
branch as the exit that makes the job green, that branch has to fall
through to a written non-zero exit, no other green exit may leave the
step, the step may not be marked unable to fail, and the write may not
appear anywhere else. A write under a condition of its own records the
decision on some runs and not others; an exit that escapes the branch
turns a red the gate REFUSED to soften into a green job with nothing
recorded; and a write on the passing path would report a suite that
passed as one that missed the deadline. What the guard cannot do is
stated with it: it cannot tell one reporter from two, it reads the
branch as text rather than evaluating it, and every block it reads is
delimited by indentation.

What is given up is real. A regression whose only symptom is that
workers never reach Ready no longer blocks a merge, and these two suites
are the only ones in the tree that boot tenant worker nodes. The
mechanism sees a timer, not a cause, so a deterministic node-join
regression rides soft as readily as a slow runner does, and the failed
job stops being the thing that opens a debugging path on its own. That
is a debt rather than a design, and the way out is written down beside
the convention it bends: calibrate the runner canary against real lane
data, then move the tolerance under the canary's alert so the lane
softens a run measured as degraded and blocks one measured as healthy.
Until that calibration exists the canary cannot carry the decision,
which is why it does not carry it today.

The deadline goes from 18m to 29m, and the reason is bounded on purpose.
29m is this test's own budget and nothing else: its clock starts where
the wait starts, after the machinedeployment wait, the LoadBalancer
assignment and the healthz probe, each with a ceiling of its own, while
the chart's timers have been running since the Machine was created. The
autoscaler's `maxNodeProvisionTime` measures from that creation and the
MachineHealthCheck's `nodeStartupTimeout` from the later of creation and
`InfrastructureReady`, restarting at that transition, so the offset is
not fixed and no ordering against either timer is claimed. What the
chart's 20m and 30m give is a scale to pick against: a budget over the
first leaves room for a remediation cycle to fit inside the window,
which is a possibility rather than a guarantee, since a run can spend
the whole window on one worker that never comes back.

The operation ceiling both suites give the script rises with the
deadline (50m to 67m), and the diagnostics phase budget rises inside it
(7m to 8m), because the collectors gated ahead of the serial console
cost more between them than the old budget covered. Both budget guards
read their terms from the sources that set them rather than restating
them. The e2e job cap rises from 180 to 215 minutes in all four lanes,
and its guard is honest about what it can hold: it derives the pair's
operation and catch ceilings from source and requires one figure across
every lane that runs the suite, and it records that the suites' full
summed ceilings (226 minutes counting the `finally` legs and the
storageclass-fallback Test) have never fit any cap in that file,
including at 180. The margin above the derivable bound is empirical
instead: three nightlies that failed both bringup Tests ran their
full-suite e2e job in 149, 158 and 160 minutes under the previous
ceilings, a run of that shape gains at most 12 minutes per suite here,
and 215 clears the resulting projection by about 30 minutes where 180
cleared its measurement by 20. The prose that restates the cap outside
the workflows moves with it, in the Chainsaw config and in the two
report collectors that derive their own headroom from it.

Test coverage: a new unit suite for the gate pins the direction it fails
in (an unreadable report, a failed testcase the report leaves unnamed or
names with the empty string or with blanks only, a failure without a
marker, and a report recording no failure all block), the lane split in
both directions, the four-link handover of the softening decision, and
the job cap. The fixtures put the unnameable case ahead of a named suite
on purpose, because an awk record runs to the next testcase and a case
placed last cannot exercise the leak the tally exists to catch. The
existing node-join suite gains guards that the marker is written only
under the status meaning the deadline expired, that it is written ahead
of the diagnostics phase rather than behind it, and that the deadline
quoted in prose across the tree is the one the wait gives, in words as
well as in minutes. Every one of those guards was checked by mutating
the thing it protects and confirming it, and only it, goes red.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
test(e2e): the tenant node-join deadline still fails its suite and still collects its diagnostics, but no longer blocks a merge in the pull-request lanes; nightly and the release-tag lane stay hard
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- E2E checks can now report a successful, non-blocking status when the
only failure is the recognized tenant node-join deadline.
- Failure diagnostics and suite results remain available for these
tolerated failures.

- **Bug Fixes**
- Improved detection and attribution of node-join deadline failures,
avoiding false-positive soft passes.

- **Documentation**
- Updated E2E timing, diagnostic budgets, and failure-handling guidance
to reflect the longer test windows.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-21 12:57:22 +03:00
Aleksei Sviridkin ad62a8d62b test(e2e): stop blocking merges on the tenant node-join deadline
The tenant node-join wait no longer discriminates. Nested
virtualisation on the shared runners degrades far enough that a guest
which registers in minutes on real hardware can miss any deadline this
test can afford, so a red on that one wait says as much about the
machine the run landed on as about the product, and it teaches a
reader to re-run the job.

The suite still fails on it. That is what keeps Chainsaw's catch
running: the previous-instance container logs, the host snapshot, the
data-plane capture and the event dump are all keyed on the test
failing, and the report stays honest. What changes is one layer up.
The wait writes a marker beside its diagnostics, and a new lane-side
gate reads that marker to decide whether the job blocks. It opens only
when every failed test in the run carries one, and blocks on anything
it cannot attribute: a report it could not read, a failed test the
report does not name, a failure with no marker.

Only the lanes that block a merge ask it. The two pull-request lanes
do; nightly and the tag lane run the same suites and stay hard. A soft
pass on the tag lane would ship a release candidate whose tenant
clusters brought up no worker nodes, and nightly is the only place an
e2e failure reaches anyone who did not open a pull request, since no
lane in this tree notifies anywhere.

A softened red finishes the job green, so the job's result stops
answering whether the suite passed, and both kinds of consumer are
handed the decision instead of inferring it. A step gated on the job
failing would silently stop running: the SSH breakpoint is that case,
and it still opens for a pull request marked for debugging. A job that
reports the result would silently start reporting a pass: the required
commit status now says the deadline was missed and that the merge is not
blocked on it, rather than that the suite passed.

The deadline goes to 29m. That is the test's own budget and nothing
more: its clock starts after the machinedeployment wait, the
LoadBalancer assignment and the healthz probe, while the chart's
autoscaler and health-check timers have been running since the Machine
was created, so no ordering against them is claimed. The chart's 20m
and 30m give a scale to pick against, and a budget over the first
leaves room for a remediation cycle without promising that one fits.

The operation ceiling both suites give the script rises with the
deadline, and the diagnostics phase budget rises inside it, because
the collectors gated ahead of the serial console cost more between
them than the old budget covered. Both budget guards read their terms
from the sources that set them. The e2e job cap rises with them, and
its guard holds what a guard can hold: the pair's operation and catch
ceilings, and one figure across every lane that runs the suite. It
also records that the suites' full summed ceilings have never fit any
cap in that file, so the margin above the bound is taken from a
measured run instead. The prose that restates the cap elsewhere moves
with it, and the sweep that holds the deadline's restatements now
reads the figure spelled in words as well as in minutes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-21 12:52:42 +03:00
Andrei Kvapil bfd4ea0643 feat(dashboard): land on the Console instead of the Marketplace
Opening the dashboard without a path sent the user to the catalog, and
the header gave Marketplace the primary emphasis. The Console is the
better entry point: it shows what the tenant already runs, which is
what someone opening the dashboard almost always came for. The
Marketplace stays one click away for when the intent is to add
something new.

The landing path becomes a named constant beside the portal helper, so
the redirect and the tab emphasis cannot drift apart, and the library's
own default tab set is reordered to match rather than contradict its
only consumer.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-21 10:27:40 +02:00
IvanHunters a00421aa17 feat(kubernetes-api): warn on Phase 2 removed fields and document them
The Kubernetes CR still accepts and stores spec.nodeGroups, spec.nodeHealthCheck
and spec.maxNodeProvisionTime after the Phase 2 split (the adoption migration
leaves them on an upgraded CR rather than rewriting parent values mid-upgrade),
but they no longer render anything, so an operator scaling via
spec.nodeGroups.<x>.minReplicas silently got nothing.

Emit a non-blocking client-facing admission warning on Create/Update when any
of these fields is present, pointing the operator at the KubernetesNodes
resources. A hard rejection is intentionally avoided: it would break edits to
already-upgraded clusters that still carry the fields. Also enumerate all the
Phase 2 breaking changes (three removed fields, two KubernetesNodes
immutability rules, the 32-char cluster-name cap) in the chart README.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:40:44 +03:00
IvanHunters 69da032103 docs(kubernetes): point references at migration 56 after the renumber
The merge renumbered the worker-pool adoption migration from 54 to 56, but
six references still named 54 (README, pre-delete-unpin hook, cluster.yaml,
worker_adoption_guard test), which now points at main's redis-operator
adoption migration. Repoint them at 56.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:35:05 +03:00
IvanHunters 7a9226da15 test(kubernetes): make control-plane-only negative asserts actually check
The 'emits no worker-pool objects' assertions used 'not: true' with
'documentIndex: -1', which scopes the check to the last rendered document
only, so a stray MachineDeployment/KubevirtMachineTemplate/MachineHealthCheck/
TalosConfigTemplate anywhere else passed silently. Move 'any: true' inside
the containsDocument matcher and drop documentIndex so the negation covers
every document. Verified the asserts now redden when a worker object is
present.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:35:05 +03:00
IvanHunters 5ca679f023 fix(kubernetes-api): cap Kubernetes cluster name so worker pools can render
A Kubernetes cluster's worker pools are separate KubernetesNodes releases
named "kubernetes-nodes-<cluster>-<pool>". validateNameLength only checked
the parent's own "kubernetes-" prefix budget (42 chars), so a cluster name
of 33-42 chars was admitted while every child pool release overflowed the
53-char Helm limit and could not render -- the migration then pinned and
skipped the pool, leaving the cluster with no way to add workers. Pre-split
nodeGroups had no such constraint, so this was a regression.

Add a kind-specific 32-char cap for the Kubernetes kind so the overflow
fails at admission on the parent CR instead of at render time on an
un-creatable child.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-21 00:35:05 +03:00
Andrei Kvapil cfd5444f16 fix(dashboard): let a masked secret be copied and confirm the copy
Copying a secret key required revealing it on screen first, and the
button gave no sign that anything reached the clipboard. The value the
list already carries is enough to copy, so the button works while the
key stays masked, and it briefly turns into a check mark to confirm.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:16:13 +02:00
Andrei Kvapil e75500682c fix(dashboard): tie navigation to where a page was opened from
Detail pages offered a fixed tab set, so Info showed Workloads, Services
and Ingresses tabs it never fills; tabs now appear only for resource
groups the instance actually owns. The probes reuse the tabs' own list
refs and label selector, so they share one cache entry and one watch.

Back and the sidebar followed the plural instead of the entry point: a
module page sent the user to a one-item list and lit nothing in the
sidebar. Module pages now return to Modules and keep it highlighted,
while Info returns to Tenants and highlights that, matching the trees
they are reached from.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:15:57 +02:00
Andrei Kvapil 1279652b56 feat(dashboard): list External IPs across every accessible tenant
The page showed LoadBalancer services of the selected tenant only, so an
operator had to switch tenants one by one to find an address. It now
lists them for every visible TenantNamespace in a single table with a
Namespace column. A namespace whose services the user cannot list
contributes no rows instead of failing the page.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:15:39 +02:00
Andrei Kvapil 21b8a0e2f0 feat(dashboard): rebuild Modules as a tenant tree of module chips
The Modules page listed the selected tenant's modules as Enabled or
Disabled cards, so a module a parent tenant provides looked disabled and
clicking it led to a Tenant CR that does not exist in the child's
namespace — an endless spinner.

It now covers every tenant the user can access, as the same hierarchy
tree the Tenants page draws, with each tenant's modules as chips
carrying the module icon, the namespace running it and its readiness.
Availability comes from the `namespace.cozystack.io/<module>` label on
each TenantNamespace, which names the providing namespace: the tenant
itself for a local module, an ancestor for an inherited one. An
inherited chip appears only when that ancestor is out of the user's
reach, since otherwise its own chip already shows the module. Clicking
any chip opens the module in whichever tenant runs it.

Tenants whose whole subtree has no modules collapse into a
"+ N tenants" branch, and the Info module is left out entirely as every
tenant carries it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:15:26 +02:00
Andrei Kvapil 7a48d455ac feat(dashboard): show the tenant hierarchy on the Tenants page
The flat table becomes a tree built from TenantNamespaces, whose
`tenant.cozystack.io/<ancestor-ns>` labels carry the full ancestry. A
tenant whose parent is not accessible attaches to its nearest visible
ancestor instead of disappearing, and rows are labelled relative to that
parent, so a sub-tenant reads as `crpjxhwm` rather than
`whmcs-crpjxhwm`. Subtrees collapse from the row itself.

Each row also carries its own actions: Info opens that tenant's Info
page, and Create Tenant orders a sub-tenant at any level of the tree.
Editing a tenant switches to the namespace holding its Tenant CR — the
parent's, or its own for the hierarchy root.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:15:07 +02:00
Andrei Kvapil ef73a0e47c feat(dashboard): move Administration into the Admin portal
Administration (Tenants, Modules, External IPs, per-tenant Info) leaves
the Console sidebar for Admin, which is now always visible: those pages
need no special permission. The two cluster-wide operator areas keep
their own gates, so Capacity and Backup Classes still appear only for
users who can open them.

Resource routes become portal-aware through a shared base-path helper,
so a detail, edit or order page opened from Admin stays under /admin
instead of jumping back to /console.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:14:50 +02:00
Andrei Kvapil 5178e169b1 feat(dashboard): drop the Marketplace landing page
Open the grouped application list right away instead of the
"Choose a product to deploy" page and its Show-all-apps button.
Category links keep filtering to a single category.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 21:14:33 +02:00
Aleksei Sviridkin 9c7bfb830b ci(workflows): make the E2E breakpoint fully label opt-in (#3926)
## What this PR does

The SSH debug breakpoint that opens on a failed end-to-end run used to
open automatically for pull requests authored by any project maintainer,
even when nobody intended to use it, in addition to opening for pull
requests carrying an explicit label. Every red end-to-end run on such a
pull request held an ephemeral CI runner open for up to twenty minutes
for a breakpoint nobody attached to, which adds up across an ordinary
day of pull requests. This change makes the breakpoint strictly opt-in:
it now opens only when the pull request carries the debug label or the
release label. A maintainer who wants to attach adds the debug label
before or after the run fails. The step's explanatory comment is trimmed
to match: the paragraph describing the removed authorship path,
including a claim about runner selection that no longer applied to the
current workflow, is removed rather than rewritten.

### Screenshots


Not applicable: this is a CI workflow text change with no user
interface.

### Downstream repositories


- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note


```release-note
NONE
```
2026-08-20 18:32:27 +03:00
Aleksei Sviridkin 61f7939ab7 ci(workflows): make E2E breakpoint opt-in via labels only
Drop the author-association branch from the "Breakpoint on E2E failure"
step condition, so a red end-to-end run only opens an SSH breakpoint
when the pull request carries the debug or release label. The dropped
branch held an ephemeral CI runner open for up to twenty minutes on
every red maintainer-authored pull request, whether or not anyone
intended to attach.

Trim the step's explanatory comment to match: drop the paragraph about
the removed authorship path, including its stale claim that only the
debug label affects runner selection, since this workflow has no
conditional runs-on to begin with.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-20 18:30:21 +03:00
Aleksei Sviridkin 4ebf9c25b4 test(e2e): capture CertificateSigningRequests in cozyreport (#3925)
## What this PR does

hack/cozyreport.sh never captured CertificateSigningRequests, so the
hypothesis that a node join hangs on kubelet certificate issuance
(bootstrap token, CSR, approval) had no artifact to check it against.
This adds a bounded CSR list and a per object yaml capture for the host
cluster, next to the existing node walk, plus the same capture for every
tenant Kubernetes cluster whose kubeconfig a failed kubernetes-*
Chainsaw test left on disk under tenantkubeconfig-<test>.

Both captures reuse the existing bounded read helpers
(cozyreport_read_object, cozyreport_select_objects), so a cut off or
refused read leaves the same truncation and failure markers every other
capture in the file already produces, and an empty CSR list is not
treated as an error. The tenant walk globs tenantkubeconfig-* and skips
silently when nothing matches, which is the ordinary case for every
suite that is not kubernetes-* and for a kubernetes-* run that passed
and cleaned up after itself.

Neither walk carries a count cap or an object time budget, unlike the
VM/VMI/service walks in the same section: this follows the file's own
documented convention that an uncapped per-object walk is acceptable
when the collection step's own ceiling is the backstop, and CSR count on
a freshly provisioned e2e cluster is bounded by node count and kubelet
retry frequency rather than by workload scale; if a real collected
report ever shows the CSR walk taking a meaningful share of collection
time, add the existing count-cap-plus-time-budget pattern here and
update the header's BUDGETS arithmetic to match.

A bats test pins the shape of both walks against the script source, and
a second test proves the tenant walk never invokes kubectl when no
tenant kubeconfig is present on disk.

### Screenshots

Not applicable, this changes an e2e diagnostic collector script, not the
UI.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```
2026-08-20 17:46:30 +03:00
Aleksei Sviridkin e6be333c06 test(e2e): capture CertificateSigningRequests in cozyreport
hack/cozyreport.sh never captured CertificateSigningRequests, leaving the
hypothesis that a node join hangs on kubelet certificate issuance
(bootstrap token, CSR, approval) unverifiable from the collected
artifact.

Add a bounded CSR list and a per object yaml capture for the host
cluster, next to the existing node walk, and the same capture for every
tenant Kubernetes cluster whose kubeconfig a failed kubernetes-*
Chainsaw test left on disk. Both walks reuse the existing bounded read
helpers, so a cut off or refused read produces the same truncation and
failure markers every other capture in the file already writes, and an
empty CSR list is not treated as an error.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-20 17:39:36 +03:00
Andrei Kvapil 2eb5374e90 fix(kubeovn-webhook): make the observation unforgeable and stop advertising an enable path that does not exist
Review found the observation could be silenced and the feature could not
be turned on. Both are fixed here; enforcement itself stays out of scope
for this step.

Rate-limit the observation instead of remembering issuers. Keying on the
issuer was the mistake: with RequestClientCert that string comes from the
caller, so a bounded set of first-seen values can be filled with 32
fabricated issuers before the API server's first call, permanently
suppressing the signal the feature exists to produce, while an unbounded
one grows without limit. A rate limit cannot be starved -- a burst costs
the current interval and the next real handshake is logged in the one
after. The two outcomes are limited separately so a flood of one cannot
hide the other, and both lines now carry the peer address, since 'none
presented' fires for any certless caller and was not attributable.

Say 'verified' rather than 'UNVERIFIED' once enforcement is on, where the
handshake really has checked the chain.

Stop telling operators to point --client-ca-file at an in-pod path. No
such path exists: the pod mounts only its own serving secret, whose
ca.crt is this webhook's serving CA, so every value either aborts startup
or rejects the API server -- and with failurePolicy: Fail that stops pod
creation cluster-wide. Say what is actually true and defer enabling to
the step that adds the volume.

Correct the claim in cert.go that this webhook configures no ClientCAs.
It can now, that pool is read once at startup and never reloaded, and the
comment justified skipping reload on the old premise. Record that
reloading has to land together with the mount, since neither is usable
alone.

Cover the observation with tests: rate limiting, the burst it exists to
survive, concurrency, both branches, the enforcing wording, and that
admission is never blocked by logging.

Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 15:51:53 +02:00
Andrei Kvapil 07fd4d9a2c fix(kubeovn-webhook): bound the issuer observation and mark it untrusted
Both points came out of review of this branch, and both live in the
observation path added for the staged rollout.

The set of seen issuers was unbounded. With tls.RequestClientCert the
server accepts whatever certificate a caller offers, so the issuer string
is supplied by that caller: anyone able to open a connection could mint
fresh issuers and grow the map and the log without limit. Cap it, and say
once when the cap is reached -- a cap hit on a quiet cluster is itself the
signal that the issuers are fabricated.

The log also invited the operator to pin '--client-ca-file' to whichever
CA it had just printed. That CA is unverified by construction, so an
attacker who can reach the port can propose one. Say plainly in the log
and in the comment that the value is a claim rather than an identity, and
that it has to be confirmed out of band. Enforcement still depends only
on --client-ca-file, which is trusted control-plane configuration.

Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-08-20 14:34:17 +02:00