cozystack/GENERAL_TECHNICAL_REVIEW.md

General Technical Review - Cozystack / Incubation

• Project: Cozystack
• Project Version: v1.6.3 (latest stable at the time of submission, released 2026-09-04). The authoritative list of releases is the GitHub Releases page; which release lines receive security fixes is defined in SECURITY.md.
• Website: https://cozystack.io
• Date Updated: 2026-09-17
• Template Version: v1.0
• Description: Cozystack is a free, Apache-2.0 open-source platform and framework for building clouds. It turns a set of bare-metal servers into a system with a native Kubernetes REST API for spawning Kubernetes clusters, Databases-as-a-Service, virtual machines, load balancers, HTTP-caching and other services. It is a CNCF Sandbox project and a Certified Kubernetes distribution.

Status: DRAFT — the remaining items are open due-diligence work, not unwritten answers. Every answer that repository or documentation evidence can settle has been written and verified against the tree. Most items still marked [maintainers: …] fall into three kinds that cannot be closed by editing this document alone: measurements — load/scale test results and tested maximums (nodes, tenants, VMs, PVs) with methodology, measured baseline resource overhead of the platform components, and the upgrade→downgrade→upgrade CI matrix and its results; maintainer confirmations of runtime behavior — whether platform upgrades trigger Talos node reboots, whether any core-component change requires a control-plane restart, CRD cleanup on full uninstall, and any explicit circuit-breaker usage; and organization actions — closing the SCA write-back loop and publishing the CVE-scanning pipeline's policy documents and scanning scripts from the private cozystack/security-scanner repository so the method is auditable independently of the numbers. The security-response-team rotation and diversity statement have since been surfaced into the public SECURITY.md, and the aggregate security reporting is published at docs/security/reports/. A few remaining [maintainers: confirm] markers — notably the third-party attribution-notice details in the Compliance section — are editorial and are to be resolved in place.

Notes for reviewers: answers cover Day 0 and Day 1 (required for Incubation, cumulative) plus Day 2 where it already applies. Doc links are pinned to the versioned /docs/v1.6/ tree; repo links are on main and should be pinned to the submitted tag before final submission. Version numbers are deliberately not restated in prose where a canonical source exists — the Releases page and SECURITY.md are cited instead, so the document does not drift with each release.

---

Day 0 - Planning Phase

Scope

• Roadmap / scope / maintainer ladder. Scope for mid-to-long-term features is managed in the open on the public GitHub Project board, GitHub Issues/Discussions, the CNCF Slack #cozystack channel, Telegram, and the regular Cozystack community meetings. Roles and decision-making are defined in GOVERNANCE.md: three roles — Users, Contributors, Maintainers — decide by consensus, falling back to a simple-majority vote (one vote per Maintainer). The promotion path is defined in CONTRIBUTOR_LADDER.md: Community Participant → Contributor → Reviewer → Maintainer, with concrete criteria (Reviewer: ≥15 PRs/yr, ≥20 reviews, ≥6 months, two sponsors from different employers; Maintainer: ≥6 months as Reviewer, ≥10 h/month). Promotions are done by pull request against the roster files in the repository root and .github/: .github/CODEOWNERS records the area-scoped Reviewer role and gates merges on the matching paths, and MAINTAINERS.md carries three sections — Active Maintainers, Reviewers and Emeritus Maintainers — naming people, while CODEOWNERS is the authoritative path-to-Reviewer mapping. The two sets are deliberately different: a Reviewer owns a code area, a Maintainer is responsible for the whole project.
• Target personas. (1) Platform/infrastructure engineers and SRE/DevOps teams building private or public clouds; (2) service providers / hosting companies / data centers offering IaaS, DBaaS and Managed-Kubernetes; (3) enterprises consolidating workloads on bare metal and seeking a self-hosted public-cloud alternative or a VMware replacement.
• Primary use case. Turn bare-metal servers into a multi-tenant cloud exposing a Kubernetes-native REST API for self-service provisioning of tenant Kubernetes clusters, VMs, managed databases, load balancers and caching.
• Additional use cases. Private/hybrid cloud; managed-services platform for providers; dev/test environments; AI-ready GPU infrastructure; on-prem VMware/vSphere replacement.
• Deployment model & constraints. Cozystack is primarily deployed as a full Talos-based distribution that owns the nodes (the recommended path), but it is not Talos-only: it can also be installed on generic Linux (Ubuntu/Debian/RHEL/Rocky/openSUSE) via the Ansible collection, which bootstraps k3s, or onto an existing / bring-your-own Kubernetes cluster (the generic-Kubernetes install); provider-specific guides exist for Hetzner (dedicated servers on Robot), Oracle Cloud (OCI) and Servers.com. Regardless of substrate it recommends a minimum of 3 servers, targets x86_64 (no Windows nodes), and single-node/2-node production is discouraged.
• Organizations that benefit. Hosting/cloud providers, data centers, telcos, financial services, and any platform-engineering organization needing sovereign, self-hosted cloud infrastructure.
• End-user research. Public adopters are listed in ADOPTERS.md. A published case study of production use is Protofire — operating Kubernetes with Cozystack; see also the case-study index and the CNCF Sandbox announcement. No formal end-user survey has been published.

Usability

• How personas interact. Administrators install and operate the management cluster; tenants/users interact declaratively via the Kubernetes API (kubectl apply of Cozystack application resources such as Kubernetes, VMInstance, Postgres, Bucket, Tenant, …), via the web dashboard, or via any Kubernetes-compatible tooling/CI, because every object is a native Kubernetes resource.
• UX/UI. Primary interface is the Kubernetes API surfaced through Cozystack's aggregated API layer (see API Design), so standard Kubernetes UX applies (kubectl, GitOps, client libraries). A web dashboard provides self-service creation/monitoring of applications and tenants, and shipped Grafana dashboards provide observability.
• Integration in production. Applications are Kubernetes CRs backed by Helm releases reconciled by Flux CD, so Cozystack integrates natively with GitOps pipelines, CI/CD and any operator ecosystem. Tenant Kubernetes clusters are standard Certified Kubernetes clusters usable with the entire CNCF landscape. Federated auth via Keycloak (OIDC/SAML) — see OIDC docs.

Design

• Design principles / best practices. Everything-is-a-Kubernetes-resource (infra, services and apps modeled uniformly as CRs and reconciled by controllers via GitOps); immutable infrastructure on the recommended path (Talos Linux, no SSH/no shell on nodes — the generic-Linux and bring-your-own-Kubernetes installs run on a conventional OS instead); composition of established CNCF/upstream building blocks rather than re-implementation; secure-by-default multi-tenancy (namespace/Tenant isolation, network policies, isolated tenant API servers); declarative, versioned Helm-based packaging.
• Architecture requirements & environments. A management cluster (the platform) provisions tenant clusters (Kubernetes-in-Kubernetes: Kamaji runs tenant control planes as pods, Cluster API with Kamaji + KubeVirt providers manages lifecycle, worker nodes/VMs on KubeVirt). Hardware tiers (hardware-requirements): Minimal/PoC 3 nodes × 8 cores / 24 GB / 50 GB + 256 GB SSD / 1 Gbps (1–2 tenants, ≤3 clusters); Small production 3 nodes × 16–32 cores / 64 GB / 100 GB + 1–2 TB (5–10 tenants); Large production 6+ nodes × 32–64 cores / 128–256 GB / 200 GB + 4–10 TB NVMe / 10 Gbps (20+ tenants). Low latency between nodes is required; VM-based nodes need nested virtualization and CPU model host.
• In-cluster service dependencies. Full platform stack (platform-stack): the node OS (Talos Linux on the recommended path; generic Linux with k3s, or an existing Kubernetes cluster, on the other paths), Kubernetes; networking Cilium (eBPF CNI + NetworkPolicy), Kube-OVN (VPC isolation, floating IPs, tenant segmentation), MetalLB, ingress-nginx, Multus CNI, CoreDNS, ExternalDNS, Kilo (WireGuard mesh); virtualization KubeVirt + CDI; storage LINSTOR/DRBD via Piraeus, SeaweedFS (object/Bucket), Velero (backup/restore), CSI Driver NFS; GitOps/automation Flux, Aenix etcd Operator (dedicated etcd for tenant control planes), cert-manager, External Secrets Operator; observability VictoriaMetrics + VictoriaLogs, Grafana, Alerta, VMAgent, Fluent Bit, kube-state-metrics, node-exporter; autoscaling Vertical Pod Autoscaler, Cluster Autoscaler; databases PostgreSQL, MariaDB, MongoDB (Percona), ClickHouse, OpenSearch, Qdrant, FoundationDB, Redis; messaging Kafka (Strimzi), NATS, RabbitMQ; identity/registry Keycloak, Harbor, OpenBao.
• Identity & Access Management. Kubernetes-native RBAC end-to-end; multi-tenancy via the Tenant resource with per-tenant namespaces, scoped RoleBindings, NetworkPolicies and quotas; each tenant Kubernetes cluster has a fully isolated API server (separate Kamaji control plane + dedicated etcd). Federated SSO via Keycloak (OIDC/SAML). Secrets integration via External Secrets Operator and OpenBao (PKI/dynamic secrets).
• Sovereignty. Fully self-hosted on the operator's own hardware; no dependency on any public cloud or external SaaS control plane — all data and control planes remain on-premises. This is a core design goal.
• Compliance. Apache-2.0; ships a Security Self-Assessment (see Security) and OpenSSF Best Practices + Scorecard. Cozystack provides isolation/RBAC/audit/network-policy/PKI primitives adopters use to build compliant platforms rather than claiming formal certifications itself. Per the self-assessment, the project makes no formal compliance attestation against any framework (PCI-DSS, ISO 27001, SOC 2, GDPR). It does publish requirement-by-requirement compliance mapping pages — PCI-DSS, GDPR, CIS Benchmark and DORA — which map platform capabilities to each framework's controls; these are guidance for adopters building compliant platforms, explicitly not attestations or certifications of Cozystack itself.
• High Availability. Management control plane HA (≥3 nodes, dedicated etcd); tenant control planes (Kamaji) run as replicated pods and reschedule on failure; storage synchronously replicated (DRBD, 2–3 replicas); MetalLB failover for LB IPs; ingress replicated.
• Resource requirements (CPU/Network/Memory). See hardware tiers above — from 3 × (8 cores / 24 GB) minimal to 6+ × (32–64 cores / 128–256 GB) at scale; 1 Gbps minimum, 10 Gbps recommended for production.
• Storage requirements. Persistent replicated block storage (LINSTOR/DRBD via Piraeus) for stateful workloads (DB and VM disks) and object storage (SeaweedFS, S3-compatible) for the Bucket service/backups; Velero for backup/restore; CSI Driver NFS for NFS-backed volumes. Nodes need a dedicated raw secondary disk for LINSTOR (single-disk is possible but discouraged).
• API Design:
  • Topology & conventions. Cozystack exposes applications through a Kubernetes API-aggregation layer: a dynamically-generated aggregated API server surfaces each Cozystack "application" as a first-class, namespaced Kubernetes resource under Cozystack API groups (apps.cozystack.io), each backed by an ApplicationDefinition and a Helm chart/HelmRelease reconciled by Flux. Conventions follow Kubernetes API machinery (group/version/kind, spec/status). Reference: REST API, Go Types, ApplicationDefinition; background: blog "How we built a dynamic Kubernetes API server for the API aggregation layer in Cozystack".
  • Defaults. Secure/production-oriented defaults: RBAC on, per-tenant isolation, network policies, replicated storage, sane per-app sizing overridable via the resource spec.
  • Additional configuration. Cluster-wide settings via the cozystack ConfigMap (bundle/variant, pod/service CIDRs, root host, API endpoint, storage) plus a MetalLB address pool; per-app tuning via the resource spec.
  • New/changed API types & cloud-provider calls. Introduces its own CRDs/aggregated types (apps.cozystack.io, Tenant, per-service kinds). Makes no external cloud-provider API calls — everything is on-prem.
  • Compatibility with the Kubernetes API server. Integrates via standard Kubernetes API-aggregation (APIService); does not fork/modify core kube-apiserver; tenant clusters run upstream Certified Kubernetes.
  • Versioning & breaking changes. API groups are versioned per Kubernetes conventions; the Cozystack application API group (apps.cozystack.io) is currently v1alpha1. No formal API stability / deprecation policy is published yet — publishing one (alpha/beta/stable lanes and a deprecation window) is outstanding. Breaking changes are communicated through release notes and changelogs.
• Release process. Semantic Versioning; releases on GitHub (stable vX.Y.0, patch vX.Y.Z). Artifacts are container images and Helm charts distributed via OCI (ghcr.io/cozystack/cozystack). Component versions are pinned per Cozystack release so an upgrade is a single coordinated bump. Only Stable/Patch releases are recommended in production (avoid alpha/beta/rc).

Installation

• How it is installed/initialized. (1) Stand up the management Kubernetes cluster via one of the supported paths (install overview): Talos (recommended) — provision bare-metal/VM nodes with Talos and bootstrap Kubernetes (Talos install); generic Linux — the Ansible collection prepares Ubuntu/Debian/RHEL/Rocky/openSUSE and bootstraps k3s; bring-your-own Kubernetes — install onto an existing cluster; plus provider guides (Hetzner dedicated/Robot, OCI, Servers.com) and an air-gapped variant. (2) Install Cozystack on that cluster in one of two modes (install): As a Platform (pick a variant such as isp-full; Cozystack installs and configures all components) or BYOP (install the operator with the default variant that provides the package registry PackageSources, then use the cozypkg CLI). The operator is delivered as an OCI Helm chart: helm ... oci://ghcr.io/cozystack/cozystack/cozy-installer. (3) Configure via the cozystack ConfigMap (variant, pod/service CIDRs, root host, API endpoint, storage) and create a MetalLB IP pool.
• How an adopter tests/validates. Verify nodes Ready (kubectl get node), all platform HelmReleases reconciled (kubectl get hr -A — none failing), core pods Running, the aggregated API responds and the dashboard is reachable; then smoke-test by provisioning a sample tenant/app (e.g. a small Kubernetes cluster or Postgres) to Ready. Tenant clusters can be validated with upstream conformance (Sonobuoy) since Cozystack is Certified Kubernetes.

Security

• Security self-assessment. Cozystack has completed a Security Self-Assessment submitted with the Incubation application (cncf/toc#1916), reviewed with TAG Security. The assessment is maintained in-tree at docs/security/self-assessment.md, alongside a threat model and an incident-response process. Where this questionnaire and the self-assessment overlap, the self-assessment is the more detailed source and the two are kept consistent.
• Cloud Native Security Tenets.
  • How satisfied: secure-by-default (RBAC, per-tenant NetworkPolicies, isolated tenant API servers, immutable Talos nodes with no SSH/shell, minimal host attack surface); defense-in-depth via network segmentation (Cilium/Kube-OVN); automated TLS via cert-manager; secret management via External Secrets Operator/OpenBao; least-privilege service accounts for controllers; supply-chain hygiene via CodeQL, OpenSSF Scorecard and organization-wide Trivy CVE scanning of dependencies and container images.
  • Loosening defaults: isolation between unrelated tenants is enforced by Cilium egress by design, and there is no supported mechanism to disable tenant isolation wholesale or to create flat/shared inter-tenant networking. What is documented is opt-in relaxation to specific restricted services: per the tenants guide, labels such as policy.cozystack.io/allow-to-apiserver: "true" and policy.cozystack.io/allow-to-etcd: "true" let tenant pods reach otherwise-restricted endpoints, and broader RBAC can be granted for single-tenant/lab use. (The earlier isolated tenant field was removed in favor of these opt-in policy labels.)
• Security Hygiene.
  • Practices: GitHub review with required approvals; CI on every PR. The branch protection on main currently requires two status checks, pre-commit and E2E Tests. CodeQL static analysis runs on every PR, on push to main and weekly; new error-severity alerts are expected to be resolved or explicitly dismissed before merge, but CodeQL is not yet a required check — promoting it to one is an open item, and SECURITY.md states it as an intention rather than a fact. zizmor audits the GitHub Actions workflows themselves and fails on findings such as unpinned actions or over-broad permissions; it runs two ways, as a hook inside the required pre-commit check (so it genuinely gates merges) and as a standalone workflow triggered only by changes under .github/workflows/ or to .github/zizmor.yml, which is not itself a required check. OpenSSF Scorecard runs weekly and publishes to scorecard.dev; the OpenSSF Best Practices badge is maintained. Organization-wide Trivy scanning covers dependencies and container images on a 6-hour cycle for CRITICAL findings and weekly for the rest, with per-CVE triage tracked as issues (see Day 2 → Dependencies). Renovate automates dependency updates within the scope described in Day 2 → Dependencies. Component versions are pinned per release.
  • Unmaintained-feature risk: components are curated and version-pinned per release; the Helm-packaged, swappable building-block design makes replacing a risky/unmaintainable component tractable. GOVERNANCE.md documents how bundled third-party components are sourced and held (vendored by digest under the upstream owner's namespace, not re-released by the project). A formal governance workflow for deciding to retire or replace a risky/unmaintained bundled component is not yet documented.
• Cloud Native Threat Modeling.
  • Least privilege: controllers run with scoped RBAC; elevated privileges are limited to and documented for infrastructure components that genuinely need them (KubeVirt/CDI for virtualization, Piraeus/LINSTOR for block devices, Cilium/Kube-OVN for CNI). The published threat model is docs/security/threat-model.md.
  • Certificate rotation: automated via cert-manager (issuance/renewal/rotation) for platform TLS; Talos/Kubernetes-managed PKI for cluster components; tenant control-plane certs managed by Kamaji.
  • Secure software supply chain: in place today — CodeQL SAST (PRs, push to main, weekly), zizmor workflow auditing (gating on PRs), OpenSSF Scorecard (weekly), continuous dependency and container-image CVE scanning with Trivy across the whole organization (see Day 2 → Dependencies → SCA), Renovate-managed dependency updates, component versions pinned per release, and private vulnerability reporting with defined CVSS response targets.
  • Supply-chain gaps: artifact signing does not exist — there is no cosign or sigstore anywhere in the build tooling, and no issue tracks adding it. Build provenance is explicitly disabled: hack/common-envs.mk sets --provenance=false in BUILDX_ARGS, so the attestation that buildx would emit by default is switched off, and no SLSA provenance is published. SBOM generation is implemented but off by default: the same file plumbs SBOM=1 through to --sbom=true, attaching a CycloneDX SBOM as a buildx attestation, and defaults it to 0 pending verification of registry referrers/attestation support; the remaining work is tracked in cozystack#2937 under the CI/build/release tracking issue cozystack#3278. Turning SBOMs on by default and re-enabling provenance are therefore configuration-and-verification work rather than greenfield work; signing is not.

---

Day 1 - Installation and Deployment Phase

Project Installation and Configuration

• As in Day 0 → Installation: a management cluster (Talos on the recommended path; generic Linux/k3s or an existing Kubernetes cluster on the other paths), then Cozystack via Platform or BYOP mode (OCI Helm installer + cozystack ConfigMap + MetalLB pool). All subsequent configuration is declarative (Kubernetes resources / Flux-managed HelmReleases); Day-2 changes are made by editing resources, not imperative steps.

Project Enablement and Rollback

• Enable/disable in a live cluster. On the Talos and generic-Linux paths Cozystack is the distribution that owns the cluster, so "enablement" = installing the operator bundle on the management cluster; on the bring-your-own-Kubernetes path it is installed onto an existing cluster instead. Individual bundles/components and each tenant application are enabled/removed independently (create/delete the corresponding resource); removing a tenant application affects only that tenant, with no downtime to unrelated workloads. [maintainers: confirm whether any core component change requires control-plane restart]
• How enablement changes default cluster behavior. Installing Cozystack sets the CNI (Cilium/Kube-OVN), default StorageClass (LINSTOR), ingress, cert-manager, and the aggregated API as cluster defaults for subsequently created workloads. Cozystack expects to own the cluster, so a clean install does not disturb pre-existing unrelated workloads.
• Testing enablement/disablement. CI installs Cozystack end-to-end in ephemeral environments and provisions/destroys sample tenants and applications to validate create→ready→delete flows. Workflows: pull-requests.yaml (per-PR checks), e2e-tag.yaml (tag/release e2e), nightly.yaml (scheduled full run) and ui-test.yaml (dashboard). Test conventions are documented in docs/agents/e2e-testing.md. Current health of these workflows: the gating checks are the per-PR ones — pre-commit and E2E Tests are the two required status checks on main, and a change cannot merge without them passing. The scheduled runs are another matter. nightly.yaml has been unreliable on the scheduled full run for a month: the first scheduled run after the fix in cozystack#3332 still failed, then it passed for three days, went red for five, passed for three more ending 2026-08-01, and has failed nearly every scheduled run since, with only isolated passes on 2026-08-17 and 2026-09-03. As of 2026-09-17, 1 of the last 24 scheduled runs passed (on 2026-09-03), and every scheduled run after it has failed. The current failure has not been triaged to an issue — several e2e failures from the same window are open and may share a cause, and the older nightly item in cozystack#3278 describes the earlier never-completing failure rather than this one — so the honest statement is that the scheduled full-matrix run is unreliable and its current failure is not yet attributed. The tag-triggered e2e-tag.yaml also failed around the v1.6.1 release. Enablement and disablement coverage therefore rests on the per-PR e2e job today, and restoring the scheduled full-matrix run is outstanding work.
• Resource cleanup incl. CRDs. Deleting an application resource triggers Flux/Helm to remove its HelmRelease and owned objects; finalizers ensure dependent resources (PVCs, VM disks, LB IPs) are released. [maintainers: confirm CRD-cleanup behavior on full uninstall and any manual steps]

Rollout, Upgrade and Rollback Planning

• Compatibility with Kubernetes & frequency. Cozystack ships a curated, tested Kubernetes version per release; tenant clusters are Certified Kubernetes following upstream skew/support. Platform upgrades are delivered as new Cozystack releases. The authoritative list of what has shipped is the GitHub Releases page; which release lines are still receiving security fixes is defined as a rolling rule in SECURITY.md: the latest stable minor and the two preceding minors are supported, older 1.x minors receive critical and upgrade-blocking fixes at maintainer discretion, v0.41.x is end of life and anything earlier is unsupported. Specific version numbers are deliberately not restated here, because they drift with every release. Note that being in a supported line does not entail receiving every fix: the project has no documented rule for which supported lines a given fix is backported to, and that is recorded in SECURITY.md rather than presented as a policy.
• Upgrade procedure (upgrade docs): run the troubleshooting/health checklist; annotate the cozy-system namespace and cozystack-version ConfigMap with helm.sh/resource-policy=keep; helm upgrade cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer --version X.Y.Z (do not use --reuse-values — the chart pins OCI repo/package versions); verify pods and kubectl get hr -A.
• Rollback. GitOps/Helm-managed with versioned releases; roll back by targeting the previous release (previous HelmRelease revisions) via the same mechanism. Stateful data (databases, VM disks) persists on replicated storage across control-plane rollbacks. No dedicated rollback runbook is published, and version-skip constraints for downgrades are not documented; the upgrade docs currently cover the forward path only.
• How a rollout/rollback can fail & impact. Risks: incompatible CRD schema change, storage/CNI component upgrades requiring node reboots (on the Talos path), or a database-operator major bump. Impact is scoped to the component; running tenant workloads generally continue, though storage/CNI upgrades can transiently affect networking or volume attachment during node rolls. [maintainers: confirm whether upgrades trigger Talos node reboots]
• Metrics that should inform a rollback. Control-plane/API availability & latency, Flux reconciliation failures (kubectl get hr -A | grep -v True), DRBD/LINSTOR replication health (linstor resource list --faulty), CNI/pod-networking errors, tenant application readiness.
• Upgrade/rollback testing (upgrade→downgrade→upgrade). [maintainers: describe the CI matrix testing N-1→N upgrades and the downgrade path; link results]
• Deprecations/removals communication. Release notes/CHANGELOG (docs/changelogs/), GitHub release announcements, docs and community channels. A dedicated, documented deprecation policy/window is not yet published — it is folded into the planned API Stability Policy (see Day 0 → API Design → Versioning).
• Alpha/beta capabilities in a rollout. Only Stable/Patch releases are recommended for production; experimental bundles/apps are opt-in. There is no formal per-application maturity field (e.g. alpha/beta/stable) in ApplicationDefinition today; maturity is expressed informally in docs and release notes. Adding maturity/compatibility metadata is a roadmap (Marketplace) item.

---

Day 2 - Day-to-Day Operations Phase

Scalability/Reliability

• Increasing size/count of API objects. Scale horizontally by adding nodes (Talos or generic-Linux, per the install path) and by creating more tenants/applications (each an independent set of Flux-reconciled Kubernetes objects); storage scales by adding disks/nodes to the LINSTOR pool; Cluster Autoscaler and Vertical Pod Autoscaler assist node/pod right-sizing.
• SLOs/SLIs. The project does not publish prescriptive reference SLOs; availability and latency targets are adopter-defined, because they depend on the operator's hardware tier, storage topology and tenant density. Cozystack exposes the SLIs needed to define and measure them: management-API availability and latency, Flux reconciliation success/failure per HelmRelease, tenant control-plane (Kamaji) pod readiness, DRBD/LINSTOR replication health, storage I/O, CNI/pod-networking errors, and ingress/LB health — all surfaced through VictoriaMetrics/Grafana (see Observability).
• Operations increasing in time under existing SLIs. Provisioning large tenant clusters, VM live-migration, and DRBD full-resync after node replacement scale with size/data volume. [maintainers: confirm]
• Resource-usage increase from enabling the project. Platform components (Cilium, Kube-OVN, KubeVirt/CDI, LINSTOR, VictoriaMetrics/VictoriaLogs, Flux, DBaaS operators) consume a baseline on the management cluster; VM/DB workloads dominate at scale. The hardware tiers above bound expected footprints. [maintainers: provide measured baseline overhead of platform components]
• Conditions causing node-resource exhaustion. Many VMs/pods per node (PIDs, sockets, hugepages), inodes on storage nodes, DRBD/LINSTOR memory under heavy replication, conntrack under high connection counts. Quotas/limits, VPA and MetalLB/ingress sizing mitigate. [maintainers: confirm]
• Load testing performed & results. [maintainers: summarize scale/load tests — nodes, tenants, VMs, DB throughput — and link results; known gap to fill for Incubation DD]
• Recommended limits & how obtained. Indicative per-tier limits are published in the hardware requirements (e.g. large-production ≈ 20+ tenants, dozens of clusters, hundreds of VMs/DBs on 6+ nodes). [maintainers: provide tested maximums (nodes, tenants, VMs, PVs) and methodology]
• Resilience patterns. Controller reconcile loops with retries/back-off (Kubernetes/Flux), synchronous storage replication (DRBD), leader election, health probes and automatic rescheduling, MetalLB failover, cert-manager auto-renewal. [maintainers: note any explicit circuit-breaker usage, e.g. at ingress]

Observability Requirements

• Signals produced (monitoring docs): metrics via VictoriaMetrics (VMCluster) collected by VMAgent (Prometheus-compatible); logs via VictoriaLogs (VLogs) collected by Fluent Bit (DaemonSet); dashboards via Grafana; alerting via Alerta; plus kube-state-metrics and node-exporter. Kubernetes events and component logs in standard formats.
• Audit logging. Kubernetes API audit logging can be enabled on management and tenant control planes through the standard kube-apiserver audit policy. No default audit policy is shipped by Cozystack — no audit-policy manifest exists in any package — so audit logging is opt-in and the policy is authored by the adopter. Enabling it by default, with a documented reference policy, is an open improvement rather than an existing feature.
• Dashboards. Pre-built Grafana dashboards for cluster overview, node metrics, storage, KubeVirt/VMs and per-application health, plus user-created dashboards (Grafana Operator manages dashboards as CRDs).
• Cost/FinOps surfacing. Per-tenant/per-application resource usage is observable via metrics and Kubernetes resource accounting (a chargeback/showback basis). No billing/FinOps integration is implemented today; a Tenant FinOps API (OpenCost-compatible) is a roadmap item. Adopters integrate the exposed usage metrics with their own provider billing/portal.
• Health parameters. Control-plane/API availability, Flux reconciliation status, Talos/node health, DRBD replication state, pod/VM readiness, ingress/LB health.
• Determining the project is in use. Presence of Cozystack CRDs/aggregated APIs (apps.cozystack.io) and reconciled platform HelmReleases; existence of Tenant/application resources.
• Knowing it works for your instance. Dashboards green; all HelmReleases Ready; sample tenant/app provisions successfully; storage replication healthy.
• Per-tenant monitoring. Tenant workloads connect custom exporters via VMServiceScrape/VMPodScrape.

Dependencies

• Running services depended on. As listed in Day 0 → in-cluster service dependencies (full platform stack).
• Dependency lifecycle policy. Component versions are pinned and shipped per Cozystack release; upgrades are curated and tested together and bumped as a single coordinated release. Renovate (.github/renovate.json) automates part of this, and it is worth being exact about which part, because the automated share is smaller than the phrase "Renovate-managed" usually implies. Its enabledManagers are gomod, dockerfile, github-actions and custom.regex; the helm-values manager is disabled repo-wide, because many image references in the vendored charts are assembled at render time by the cozy-lib.image template helper rather than written literally in values.yaml, so a values-file manager cannot see them. Two bespoke regex managers cover specific pins that would otherwise age silently. Consequently: Go modules, Dockerfile base images, GitHub Actions and those regex-pinned refs get automated PRs, while the curated component images that make up the shipped platform are bumped by maintainers as part of preparing a release, not by a bot. That is a deliberate consequence of shipping a tested, coordinated version set, but it does mean the dependency class that dominates the product is on human cadence.
• Source Composition Analysis (SCA). Four layers. (1) Static analysis of first-party code — CodeQL on every PR, on push to main and weekly, reporting only alerts new relative to main; expected to be resolved or dismissed before merge, though not yet a required check. (2) Static analysis of the CI supply chain itself — zizmor audits the GitHub Actions workflow definitions for unpinned actions, over-broad permissions and similar findings; it gates merges through the required pre-commit check, with an additional workflow covering tree-wide runs when the workflow definitions themselves change. (3) Supply-chain posture — OpenSSF Scorecard weekly, published on scorecard.dev, plus the OpenSSF Best Practices badge. (4) Dependency and container-image CVE scanning — an organization-wide pipeline built on Trivy running against every non-fork, non-archived repository in the cozystack organization.
• How the CVE-scanning pipeline works. Discovery walks each repository for Go modules (go.mod, direct dependencies), base images from Dockerfile FROM directives, and container images referenced in values.yaml under packages/apps/*, packages/system/* and packages/core/*. Trivy then scans both the repository filesystem — which is where transitive Go dependencies and lock files are actually covered, since discovery itself reads only direct requirements — and every resolved container image, against NVD, vendor advisories and the GitHub Advisory Database. A known limitation: image references assembled at render time by the cozy-lib.image helper are not literal strings in values.yaml and are therefore invisible to values-file discovery, the same blind spot that led to disabling Renovate's helm-values manager. Those images are covered only insofar as they appear elsewhere; closing the gap requires resolving refs through the rendered chart rather than the values file. CRITICAL findings are scanned and reported every 6 hours; HIGH/MEDIUM/LOW are collected into a weekly report. Four filters run before an issue is created: findings with no upstream fix older than 365 days; dev and build-only dependencies (golang:/node:/python: images, core/testing, hack/, e2e/); CVEs already carrying a maintainer triage decision; and findings with no severity assigned in any database. As a result, every new actionable CVE becomes a tracked issue with severity labels, affected components, fix version and a triage checklist, while roughly 20–30% of raw output is suppressed as noise. Triage decisions are persisted, so a settled finding is not re-reported.
• Public verifiability of the pipeline — an open problem. The pipeline, its reports and its triage state live in cozystack/security-scanner, which returns 404 to anyone outside the organization, so none of the above is currently checkable by a reviewer without privileged access. The line that needs drawing is not per-repository but per-artifact, and it falls in a specific place: severity-level aggregate counts are publishable; naming an individual finding that is not yet fixed is not, because a CVE ID plus an affected package plus this project's public, version-pinned component list is enough to locate unpatched exposure in a running deployment. Against that test, the pipeline policy (SECURITY_PIPELINE_POLICY.md), the severity and SLA matrix (SECURITY_TRIAGE_MATRIX.md), the finding status model (SECURITY_FINDING_STATUS_MODEL.md) and the scanning scripts hold no finding data and are publishable as they stand — and they are the part that actually evidences the control. Monthly reports for every month the pipeline has run are now published in-tree at docs/security/reports/, with identifiers withheld unless the fix is merged and released and a generator guard that refuses to emit a file containing an unreleased one. They are accompanied by a backlog reconciliation auditing every tracked finding against what the project actually shipped. That audit revised the disclosure rule in two directions that make it more open: findings with no upstream fix will be named with mitigation guidance, since there is nothing to wait for and an adopter needs to know when a mitigation is the only available control; and remediation is no longer attributed to individual months, because a count of still-open findings crossed with one month's publicly enumerable component bumps narrows the search for unpatched exposure more than a bare count does. What remains is publishing the pipeline's own policy documents and scanning scripts, which hold no finding data. [maintainers: publish SECURITY_PIPELINE_POLICY.md, SECURITY_TRIAGE_MATRIX.md, SECURITY_FINDING_STATUS_MODEL.md and the scanning scripts, so the method is auditable independently of the numbers]
• Acting on SCA & timescale. Two tracks with distinct targets. Reported vulnerabilities (Cozystack-specific code, packaging, defaults, integration logic) follow SECURITY.md: acknowledgement ≤3 business days, initial triage ≤7 business days, remediation targets Critical ≈14 days, High ≈30 days, Medium ≈90 days, Low next release, under a coordinated-disclosure window of up to 90 days. Scanner findings in bundled upstream components are a separate track with a tighter triage clock: Critical triaged within 1 business day against a 7-day fix target, High within 3 business days / 30 days, Medium within 10 business days / 90 days, Low within 30 days best-effort. Triage follows an explicit decision tree (is the CVE in a shipped artifact rather than dev/test tooling; is the vulnerable path reachable in Cozystack's usage; has the distribution already backported the fix; is an upstream fix available) and resolves to one of confirmed, false-positive, accepted-risk, in-progress or fixed. Confirmed findings ship as a pinned-component version bump in the next Cozystack release, or as a patch release on a supported line when severity warrants it. Attainment is uneven, and an audit of the pipeline's own output found its state misleading in both directions. Detection keeps up. Triage now has an owner, and continuous attainment is what is left to demonstrate. The backlog was decided in a single bulk pass rather than continuously, because the rotating role that owns the triage clock, though defined, was unassigned; that role has now been assigned to @tym83, who runs a weekly triage pass, so sustained continuous attainment of the triage SLAs is the remaining target rather than the vacancy itself. Remediation, however, is happening and the tracker cannot see it, because nothing writes back from the main repository and so no finding has ever been marked fixed. Reconciling every tracked finding against merged changes, release tags and the pipeline's own weekly scan reports establishes two things. First, a clear majority of confirmed findings relate to a component whose in-tree version now carries the fix, concentrated in a single remediation wave in June 2026 rather than a steady stream. Second, four component bumps are available upstream and not yet taken. The same audit found that 57% of the backlog was never applicable or cannot be assessed — the largest block being findings against packages that contain no executable vulnerable code, chiefly kernel headers, since a container never provides a running kernel. Reachable exposure in a stock installation is estimated at 2–5% of the tracker. Seven defects were found in the pipeline itself, including deduplication by CVE identifier alone and a broken image-attribution step that left 16% of findings unattributable. Figures, method and defect list: backlog reconciliation. [maintainers: close the write-back loop and take the four bumps that are available upstream and not yet pulled in]

Troubleshooting

• Recovery when a key component fails (troubleshooting): Kubernetes/etcd HA (and Talos self-healing on the Talos path) recover control-plane node loss; Kamaji reschedules tenant control planes; DRBD/LINSTOR promotes a healthy replica; Flux re-reconciles drifted/removed platform objects; databases recover via their operators; Velero provides backup/restore. Diagnostics: kubectl get node; kubectl get hr -A | grep -v True; kubectl get pod -A | grep -v 'Running\|Completed'; linstor node/storage-pool/resource list [--faulty]; operator logs kubectl logs -n cozy-system deploy/cozystack-operator; dedicated guides for etcd, Flux CD and Kube-OVN.
• Known failure modes. Documented areas include broken HelmReleases, LINSTOR/DRBD faulty resources, Kube-OVN OVN-database recovery, and etcd quorum issues, each with a dedicated guide under the troubleshooting section (etcd, Flux CD, Kube-OVN, LINSTOR controller / database, monitoring). These are per-topic pages; a single consolidated "known failure modes + mitigations" index is not yet published.

Compliance

• Third-party attribution & license notices. The project is Apache-2.0 with a top-level LICENSE, and bundles third-party components under their own licenses, whose license files travel with the vendored component. There is no top-level NOTICE file. For a distribution that ships on the order of fifty upstream components, that is a real compliance gap rather than an oversight to paper over: adding a NOTICE that aggregates attribution for the bundled components, and a documented process for keeping it in step with the component set, is outstanding work.
• Alignment with CNCF attribution recommendations.
  • Notices for third-party code in source files: retained in-file / in LICENSE/NOTICE. [maintainers: confirm]
  • Notices for unmodified third-party components in the repo: retained alongside the pinned component. [maintainers: confirm]
  • Notices in distributed build artifacts (images/binaries): SBOM generation is implemented but off by default — SBOM=1 in hack/common-envs.mk attaches a CycloneDX SBOM as a buildx attestation, disabled pending registry-referrer verification (tracked in cozystack#2937); release images are not yet signed and build provenance is disabled (see Day 0 → Security → Supply-chain gaps). Bundled-component license terms are catalogued in the licenses documentation.

Security

• Access control execution. Kubernetes RBAC end-to-end; per-tenant namespaces, RoleBindings, quotas and NetworkPolicies; isolated tenant API servers (Kamaji + dedicated etcd); Keycloak OIDC/SAML for federated auth; secrets via External Secrets Operator / OpenBao.
• Security reporting team & rotation. Reporting via GitHub Private Vulnerability Reporting (advisories/new) or a private maintainer contact; any maintainer can receive and route a report. Current security response team: @kvaps, @lexfrei, @tym83, @matthieu-robin, @mattia-eleuteri — spanning more than one organization (Ænix, HIDORA) and more than one country, rather than a single employer. A rotating Security Champion role (a fixed term with a handoff overlap) owns the CVE-triage clock; the role is currently held by @tym83, who runs a weekly triage pass over scanner findings, while technical reachability of a given CVE in bundled components is assessed together with the security team. This rotation cadence and the team's diversity statement are published in the project's SECURITY.md.

---

Appendix — sources

• cozystack.io docs v1.6: platform-stack, hardware-requirements, install, upgrade, monitoring, troubleshooting, OIDC, REST API.
• Repo (main): GOVERNANCE.md, CONTRIBUTOR_LADDER.md, MAINTAINERS.md, .github/CODEOWNERS, SECURITY.md, ADOPTERS.md.
• Releases: GitHub Releases.
• CI / security workflows: pull-requests.yaml, e2e-tag.yaml, nightly.yaml, ui-test.yaml, codeql.yml, scorecard.yml, renovate.json.
• Roadmap: GitHub Project board; supply-chain items cozystack#2937, cozystack#3278.
• SCA tooling: Trivy (organization-wide CVE scanning pipeline; the repository hosting it is private, see Day 2 → Dependencies).
• SSA / Incubation: cncf/toc#1916.
• GTR template: cncf/toc toc_subprojects/project-reviews-subproject/general-technical-questions.md v1.0.