diff --git a/GENERAL_TECHNICAL_REVIEW.md b/GENERAL_TECHNICAL_REVIEW.md index 0e60bdf7a..9e7818b55 100644 --- a/GENERAL_TECHNICAL_REVIEW.md +++ b/GENERAL_TECHNICAL_REVIEW.md @@ -1,17 +1,13 @@ # General Technical Review - Cozystack / Incubation - **Project:** Cozystack -- **Project Version:** v1.x (current stable line). The authoritative list of releases is the [GitHub Releases page](https://github.com/cozystack/cozystack/releases); which release lines receive security fixes is defined in [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md). `[maintainers: pin the exact release tag being submitted]` +- **Project Version:** v1.6.3 (latest stable at the time of submission, released 2026-09-04). The authoritative list of releases is the [GitHub Releases page](https://github.com/cozystack/cozystack/releases); which release lines receive security fixes is defined in [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md). - **Website:** https://cozystack.io -- **Date Updated:** 2026-08-10 +- **Date Updated:** 2026-09-17 - **Template Version:** v1.0 - **Description:** Cozystack is a free, Apache-2.0 open-source platform and framework for building clouds. It turns a set of bare-metal servers into a system with a native Kubernetes REST API for spawning Kubernetes clusters, Databases-as-a-Service, virtual machines, load balancers, HTTP-caching and other services. It is a CNCF Sandbox project and a Certified Kubernetes distribution. -> **Status: DRAFT — work in progress.** This questionnaire is being actively completed and reviewed. Wherever repository evidence settles an answer, it has been written and verified against the tree; the remaining `[maintainers: …]` markers are of two kinds, and only the first kind blocks submission. Some editorial markers are certainly still closable from the tree. Reviewers should treat an open marker as "not yet checked" rather than "checked and unanswerable". -> -> **Substantive gaps — need measurement, publication or a decision:** (1) load/scale test results and tested maximums (nodes, tenants, VMs, PVs) with methodology; (2) whether the project publishes reference SLOs/SLIs or treats them as adopter-defined; (3) measured baseline resource overhead of the platform components; (4) the upgrade→downgrade→upgrade CI matrix and its results; (5) whether platform upgrades trigger Talos node reboots; (6) the security-response-team rotation and diversity policy — a rotating Security Champion role with a 6-month term and a handoff overlap **is** defined in the CVE-scanning pipeline's own governance document, but that document is unpublished, the current holder of the role is unassigned, and nothing about rotation or diversity appears in the public `SECURITY.md`; (7) filling the unassigned Security Champion role (which owns the triage clock, and whose vacancy is why the backlog has no continuous owner) and publishing the CVE-scanning pipeline's policy documents and scanning scripts. Publishing those is the last step before the SCA answer below becomes verifiable from outside the organization; the aggregate reporting itself is now published at [`docs/security/reports/`](https://github.com/cozystack/cozystack/tree/main/docs/security/reports). -> -> **Editorial markers** — the rest are requests to link an existing document or confirm a detail, and are resolved as the corresponding docs are located or written. +> **Status: DRAFT — the remaining items are open due-diligence work, not unwritten answers.** Every answer that repository or documentation evidence can settle has been written and verified against the tree. Most items still marked `[maintainers: …]` fall into three kinds that cannot be closed by editing this document alone: **measurements** — load/scale test results and tested maximums (nodes, tenants, VMs, PVs) with methodology, measured baseline resource overhead of the platform components, and the upgrade→downgrade→upgrade CI matrix and its results; **maintainer confirmations of runtime behavior** — whether platform upgrades trigger Talos node reboots, whether any core-component change requires a control-plane restart, CRD cleanup on full uninstall, and any explicit circuit-breaker usage; and **organization actions** — closing the SCA write-back loop and publishing the CVE-scanning pipeline's policy documents and scanning scripts from the private `cozystack/security-scanner` repository so the method is auditable independently of the numbers. The security-response-team rotation and diversity statement have since been surfaced into the public [`SECURITY.md`](https://github.com/cozystack/cozystack/blob/main/SECURITY.md), and the aggregate security reporting is published at [`docs/security/reports/`](https://github.com/cozystack/cozystack/tree/main/docs/security/reports). A few remaining `[maintainers: confirm]` markers — notably the third-party attribution-notice details in the Compliance section — are editorial and are to be resolved in place. > > Notes for reviewers: answers cover **Day 0** and **Day 1** (required for Incubation, cumulative) plus **Day 2** where it already applies. Doc links are pinned to the versioned `/docs/v1.6/` tree; repo links are on `main` and should be pinned to the submitted tag before final submission. Version numbers are deliberately not restated in prose where a canonical source exists — the [Releases page](https://github.com/cozystack/cozystack/releases) and [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md) are cited instead, so the document does not drift with each release. @@ -21,13 +17,13 @@ ### Scope -* **Roadmap / scope / maintainer ladder.** Scope for mid-to-long-term features is managed in the open on the public [GitHub Project board](https://github.com/orgs/cozystack/projects/1), GitHub Issues/Discussions, the CNCF Slack `#cozystack` channel, Telegram, and the regular Cozystack community meetings. Roles and decision-making are defined in [GOVERNANCE.md](https://github.com/cozystack/cozystack/blob/main/GOVERNANCE.md): four roles — **Users, Contributors, Directors, Maintainers** — decide by consensus, falling back to a simple-majority vote (one vote per Maintainer and Director). The promotion path is defined in [CONTRIBUTOR_LADDER.md](https://github.com/cozystack/cozystack/blob/main/CONTRIBUTOR_LADDER.md): **Community Participant → Contributor → Reviewer → Maintainer**, with concrete criteria (Reviewer: ≥15 PRs/yr, ≥20 reviews, ≥6 months, two sponsors from different employers; Maintainer: ≥6 months as Reviewer, ≥10 h/month). Promotions are done by pull request against the roster files in the repository root and `.github/`: [`.github/CODEOWNERS`](https://github.com/cozystack/cozystack/blob/main/.github/CODEOWNERS) records the area-scoped Reviewer role and gates merges on the matching paths, and [`MAINTAINERS.md`](https://github.com/cozystack/cozystack/blob/main/MAINTAINERS.md) carries three sections — Active Maintainers, Reviewers and Emeritus Maintainers — naming people, while `CODEOWNERS` is the authoritative path-to-Reviewer mapping. The two sets are deliberately different: a Reviewer owns a code area, a Maintainer is responsible for the whole project. +* **Roadmap / scope / maintainer ladder.** Scope for mid-to-long-term features is managed in the open on the public [GitHub Project board](https://github.com/orgs/cozystack/projects/1), GitHub Issues/Discussions, the CNCF Slack `#cozystack` channel, Telegram, and the regular Cozystack community meetings. Roles and decision-making are defined in [GOVERNANCE.md](https://github.com/cozystack/cozystack/blob/main/GOVERNANCE.md): three roles — **Users, Contributors, Maintainers** — decide by consensus, falling back to a simple-majority vote (one vote per Maintainer). The promotion path is defined in [CONTRIBUTOR_LADDER.md](https://github.com/cozystack/cozystack/blob/main/CONTRIBUTOR_LADDER.md): **Community Participant → Contributor → Reviewer → Maintainer**, with concrete criteria (Reviewer: ≥15 PRs/yr, ≥20 reviews, ≥6 months, two sponsors from different employers; Maintainer: ≥6 months as Reviewer, ≥10 h/month). Promotions are done by pull request against the roster files in the repository root and `.github/`: [`.github/CODEOWNERS`](https://github.com/cozystack/cozystack/blob/main/.github/CODEOWNERS) records the area-scoped Reviewer role and gates merges on the matching paths, and [`MAINTAINERS.md`](https://github.com/cozystack/cozystack/blob/main/MAINTAINERS.md) carries three sections — Active Maintainers, Reviewers and Emeritus Maintainers — naming people, while `CODEOWNERS` is the authoritative path-to-Reviewer mapping. The two sets are deliberately different: a Reviewer owns a code area, a Maintainer is responsible for the whole project. * **Target personas.** (1) Platform/infrastructure engineers and SRE/DevOps teams building private or public clouds; (2) service providers / hosting companies / data centers offering IaaS, DBaaS and Managed-Kubernetes; (3) enterprises consolidating workloads on bare metal and seeking a self-hosted public-cloud alternative or a VMware replacement. * **Primary use case.** Turn bare-metal servers into a multi-tenant cloud exposing a Kubernetes-native REST API for self-service provisioning of tenant Kubernetes clusters, VMs, managed databases, load balancers and caching. * **Additional use cases.** Private/hybrid cloud; managed-services platform for providers; dev/test environments; AI-ready GPU infrastructure; on-prem VMware/vSphere replacement. -* **Explicitly unsupported.** Cozystack is a **full distribution that owns the cluster and the nodes** (Talos-based) — it is not an add-on installed into an existing, externally-managed production Kubernetes cluster, is not a public-cloud managed-Kubernetes add-on, requires **a minimum of 3 servers**, and targets **x86_64** (no Windows nodes). Single-node/2-node production is discouraged. +* **Deployment model & constraints.** Cozystack is primarily deployed as a **full Talos-based distribution that owns the nodes** (the recommended path), but it is **not Talos-only**: it can also be installed on **generic Linux** (Ubuntu/Debian/RHEL/Rocky/openSUSE) via the Ansible collection, which bootstraps k3s, or onto an **existing / bring-your-own Kubernetes** cluster (the generic-Kubernetes install); provider-specific guides exist for Hetzner (dedicated servers on Robot), Oracle Cloud (OCI) and Servers.com. Regardless of substrate it **recommends a minimum of 3 servers**, targets **x86_64** (no Windows nodes), and single-node/2-node production is discouraged. * **Organizations that benefit.** Hosting/cloud providers, data centers, telcos, financial services, and any platform-engineering organization needing sovereign, self-hosted cloud infrastructure. -* **End-user research.** Public adopters are listed in [ADOPTERS.md](https://github.com/cozystack/cozystack/blob/main/ADOPTERS.md). `[maintainers: link any published case studies / end-user survey results]` +* **End-user research.** Public adopters are listed in [ADOPTERS.md](https://github.com/cozystack/cozystack/blob/main/ADOPTERS.md). A published case study of production use is [Protofire — operating Kubernetes with Cozystack](https://cozystack.io/blog/2025/09/protofire-experience-operating-kubernetes-with-cozystack/); see also the [case-study index](https://cozystack.io/article_types/case/) and the [CNCF Sandbox announcement](https://www.cncf.io/blog/2025/04/28/open-source-paas-cozystack-becomes-a-cncf-sandbox-project/). No formal end-user survey has been published. ### Usability @@ -37,12 +33,12 @@ ### Design -* **Design principles / best practices.** Everything-is-a-Kubernetes-resource (infra, services and apps modeled uniformly as CRs and reconciled by controllers via GitOps); immutable infrastructure (**Talos Linux**, no SSH/no shell on nodes); composition of established CNCF/upstream building blocks rather than re-implementation; secure-by-default multi-tenancy (namespace/`Tenant` isolation, network policies, isolated tenant API servers); declarative, versioned Helm-based packaging. +* **Design principles / best practices.** Everything-is-a-Kubernetes-resource (infra, services and apps modeled uniformly as CRs and reconciled by controllers via GitOps); immutable infrastructure on the recommended path (**Talos Linux**, no SSH/no shell on nodes — the generic-Linux and bring-your-own-Kubernetes installs run on a conventional OS instead); composition of established CNCF/upstream building blocks rather than re-implementation; secure-by-default multi-tenancy (namespace/`Tenant` isolation, network policies, isolated tenant API servers); declarative, versioned Helm-based packaging. * **Architecture requirements & environments.** A **management cluster** (the platform) provisions **tenant clusters** (Kubernetes-in-Kubernetes: **Kamaji** runs tenant control planes as pods, **Cluster API** with Kamaji + KubeVirt providers manages lifecycle, worker nodes/VMs on **KubeVirt**). Hardware tiers ([hardware-requirements](https://cozystack.io/docs/v1.6/install/hardware-requirements/)): **Minimal/PoC** 3 nodes × 8 cores / 24 GB / 50 GB + 256 GB SSD / 1 Gbps (1–2 tenants, ≤3 clusters); **Small production** 3 nodes × 16–32 cores / 64 GB / 100 GB + 1–2 TB (5–10 tenants); **Large production** 6+ nodes × 32–64 cores / 128–256 GB / 200 GB + 4–10 TB NVMe / 10 Gbps (20+ tenants). Low latency between nodes is required; VM-based nodes need nested virtualization and CPU model `host`. -* **In-cluster service dependencies.** Full platform stack ([platform-stack](https://cozystack.io/docs/v1.6/guides/platform-stack/)): Talos Linux, Kubernetes; **networking** Cilium (eBPF CNI + NetworkPolicy), Kube-OVN (VPC isolation, floating IPs, tenant segmentation), MetalLB, ingress-nginx, Multus CNI, CoreDNS, ExternalDNS, Kilo (WireGuard mesh); **virtualization** KubeVirt + CDI; **storage** LINSTOR/DRBD via Piraeus, SeaweedFS (object/Bucket), Velero (backup/restore), CSI Driver NFS; **GitOps/automation** Flux, Aenix etcd Operator (dedicated etcd for tenant control planes), cert-manager, External Secrets Operator; **observability** VictoriaMetrics + VictoriaLogs, Grafana, Alerta, VMAgent, Fluent Bit, kube-state-metrics, node-exporter; **autoscaling** Vertical Pod Autoscaler, Cluster Autoscaler; **databases** PostgreSQL, MariaDB, MongoDB (Percona), ClickHouse, OpenSearch, Qdrant, FoundationDB, Redis; **messaging** Kafka (Strimzi), NATS, RabbitMQ; **identity/registry** Keycloak, Harbor, OpenBao. +* **In-cluster service dependencies.** Full platform stack ([platform-stack](https://cozystack.io/docs/v1.6/guides/platform-stack/)): the node OS (Talos Linux on the recommended path; generic Linux with k3s, or an existing Kubernetes cluster, on the other paths), Kubernetes; **networking** Cilium (eBPF CNI + NetworkPolicy), Kube-OVN (VPC isolation, floating IPs, tenant segmentation), MetalLB, ingress-nginx, Multus CNI, CoreDNS, ExternalDNS, Kilo (WireGuard mesh); **virtualization** KubeVirt + CDI; **storage** LINSTOR/DRBD via Piraeus, SeaweedFS (object/Bucket), Velero (backup/restore), CSI Driver NFS; **GitOps/automation** Flux, Aenix etcd Operator (dedicated etcd for tenant control planes), cert-manager, External Secrets Operator; **observability** VictoriaMetrics + VictoriaLogs, Grafana, Alerta, VMAgent, Fluent Bit, kube-state-metrics, node-exporter; **autoscaling** Vertical Pod Autoscaler, Cluster Autoscaler; **databases** PostgreSQL, MariaDB, MongoDB (Percona), ClickHouse, OpenSearch, Qdrant, FoundationDB, Redis; **messaging** Kafka (Strimzi), NATS, RabbitMQ; **identity/registry** Keycloak, Harbor, OpenBao. * **Identity & Access Management.** Kubernetes-native **RBAC** end-to-end; multi-tenancy via the `Tenant` resource with per-tenant namespaces, scoped RoleBindings, NetworkPolicies and quotas; each tenant Kubernetes cluster has a fully isolated API server (separate **Kamaji** control plane + dedicated etcd). Federated SSO via **Keycloak** (OIDC/SAML). Secrets integration via **External Secrets Operator** and **OpenBao** (PKI/dynamic secrets). * **Sovereignty.** Fully self-hosted on the operator's own hardware; no dependency on any public cloud or external SaaS control plane — all data and control planes remain on-premises. This is a core design goal. -* **Compliance.** Apache-2.0; ships a Security Self-Assessment (see Security) and OpenSSF Best Practices + Scorecard. Cozystack provides isolation/RBAC/audit/network-policy/PKI primitives adopters use to build compliant platforms rather than claiming formal certifications itself. `[maintainers: list any published compliance mappings]` +* **Compliance.** Apache-2.0; ships a Security Self-Assessment (see Security) and OpenSSF Best Practices + Scorecard. Cozystack provides isolation/RBAC/audit/network-policy/PKI primitives adopters use to build compliant platforms rather than claiming formal certifications itself. Per the [self-assessment](https://github.com/cozystack/cozystack/blob/main/docs/security/self-assessment.md), the project makes **no formal compliance attestation** against any framework (PCI-DSS, ISO 27001, SOC 2, GDPR). It does publish requirement-by-requirement compliance **mapping** pages — [PCI-DSS](https://cozystack.io/compliance/pci-dss/), [GDPR](https://cozystack.io/compliance/gdpr/), [CIS Benchmark](https://cozystack.io/compliance/cis-benchmark/) and [DORA](https://cozystack.io/compliance/dora/) — which map platform capabilities to each framework's controls; these are guidance for adopters building compliant platforms, explicitly **not** attestations or certifications of Cozystack itself. * **High Availability.** Management control plane HA (≥3 nodes, dedicated etcd); tenant control planes (Kamaji) run as replicated pods and reschedule on failure; storage synchronously replicated (DRBD, 2–3 replicas); MetalLB failover for LB IPs; ingress replicated. * **Resource requirements (CPU/Network/Memory).** See hardware tiers above — from 3 × (8 cores / 24 GB) minimal to 6+ × (32–64 cores / 128–256 GB) at scale; 1 Gbps minimum, 10 Gbps recommended for production. * **Storage requirements.** **Persistent** replicated block storage (LINSTOR/DRBD via Piraeus) for stateful workloads (DB and VM disks) and **object storage** (SeaweedFS, S3-compatible) for the Bucket service/backups; **Velero** for backup/restore; **CSI Driver NFS** for NFS-backed volumes. Nodes need a dedicated raw secondary disk for LINSTOR (single-disk is possible but discouraged). @@ -52,12 +48,12 @@ * **Additional configuration.** Cluster-wide settings via the `cozystack` ConfigMap (bundle/variant, pod/service CIDRs, root host, API endpoint, storage) plus a MetalLB address pool; per-app tuning via the resource spec. * **New/changed API types & cloud-provider calls.** Introduces its own CRDs/aggregated types (`apps.cozystack.io`, `Tenant`, per-service kinds). Makes **no external cloud-provider API calls** — everything is on-prem. * **Compatibility with the Kubernetes API server.** Integrates via standard Kubernetes API-aggregation (APIService); does not fork/modify core kube-apiserver; tenant clusters run upstream **Certified Kubernetes**. - * **Versioning & breaking changes.** API groups versioned per Kubernetes conventions; breaking changes gated via release notes/deprecation windows. `[maintainers: confirm and link the API stability/deprecation policy]` + * **Versioning & breaking changes.** API groups are versioned per Kubernetes conventions; the Cozystack application API group (`apps.cozystack.io`) is currently `v1alpha1`. **No formal API stability / deprecation policy is published yet** — publishing one (alpha/beta/stable lanes and a deprecation window) is outstanding. Breaking changes are communicated through release notes and changelogs. * **Release process.** Semantic Versioning; releases on GitHub (stable `vX.Y.0`, patch `vX.Y.Z`). Artifacts are container images and Helm charts distributed via OCI (`ghcr.io/cozystack/cozystack`). Component versions are pinned per Cozystack release so an upgrade is a single coordinated bump. Only Stable/Patch releases are recommended in production (avoid alpha/beta/rc). ### Installation -* **How it is installed/initialized.** (1) Provision bare-metal (or nested-virt) nodes with **Talos Linux** (PXE/ISO) and bootstrap a management Kubernetes cluster ([Talos install](https://cozystack.io/docs/v1.6/install/talos/), [Kubernetes install](https://cozystack.io/docs/v1.6/install/kubernetes/)). (2) Install Cozystack on that cluster in one of two modes ([install](https://cozystack.io/docs/v1.6/install/cozystack/)): **As a Platform** (pick a variant such as `isp-full`; Cozystack installs and configures all components) or **BYOP** (install the operator with the `default` variant that provides the package registry `PackageSources`, then use the `cozypkg` CLI). The operator is delivered as an OCI Helm chart: `helm ... oci://ghcr.io/cozystack/cozystack/cozy-installer`. (3) Configure via the `cozystack` ConfigMap (variant, pod/service CIDRs, root host, API endpoint, storage) and create a MetalLB IP pool. +* **How it is installed/initialized.** (1) Stand up the management Kubernetes cluster via one of the supported paths ([install overview](https://cozystack.io/docs/v1.6/install/)): **Talos** (recommended) — provision bare-metal/VM nodes with Talos and bootstrap Kubernetes ([Talos install](https://cozystack.io/docs/v1.6/install/talos/)); **generic Linux** — the [Ansible collection](https://cozystack.io/docs/v1.6/install/ansible/) prepares Ubuntu/Debian/RHEL/Rocky/openSUSE and bootstraps k3s; **bring-your-own Kubernetes** — install onto an [existing cluster](https://cozystack.io/docs/v1.6/install/kubernetes/generic/); plus provider guides (Hetzner dedicated/Robot, OCI, Servers.com) and an air-gapped variant. (2) Install Cozystack on that cluster in one of two modes ([install](https://cozystack.io/docs/v1.6/install/cozystack/)): **As a Platform** (pick a variant such as `isp-full`; Cozystack installs and configures all components) or **BYOP** (install the operator with the `default` variant that provides the package registry `PackageSources`, then use the `cozypkg` CLI). The operator is delivered as an OCI Helm chart: `helm ... oci://ghcr.io/cozystack/cozystack/cozy-installer`. (3) Configure via the `cozystack` ConfigMap (variant, pod/service CIDRs, root host, API endpoint, storage) and create a MetalLB IP pool. * **How an adopter tests/validates.** Verify nodes Ready (`kubectl get node`), all platform HelmReleases reconciled (`kubectl get hr -A` — none failing), core pods Running, the aggregated API responds and the dashboard is reachable; then smoke-test by provisioning a sample tenant/app (e.g. a small `Kubernetes` cluster or `Postgres`) to `Ready`. Tenant clusters can be validated with upstream conformance (Sonobuoy) since Cozystack is Certified Kubernetes. ### Security @@ -65,10 +61,10 @@ * **Security self-assessment.** Cozystack has completed a **Security Self-Assessment** submitted with the Incubation application ([cncf/toc#1916](https://github.com/cncf/toc/issues/1916)), reviewed with TAG Security. The assessment is maintained in-tree at [`docs/security/self-assessment.md`](https://github.com/cozystack/cozystack/blob/main/docs/security/self-assessment.md), alongside a [threat model](https://github.com/cozystack/cozystack/blob/main/docs/security/threat-model.md) and an [incident-response process](https://github.com/cozystack/cozystack/blob/main/docs/security/incident-response.md). Where this questionnaire and the self-assessment overlap, the self-assessment is the more detailed source and the two are kept consistent. * **Cloud Native Security Tenets.** * *How satisfied:* secure-by-default (RBAC, per-tenant NetworkPolicies, isolated tenant API servers, immutable Talos nodes with no SSH/shell, minimal host attack surface); defense-in-depth via network segmentation (Cilium/Kube-OVN); automated TLS via **cert-manager**; secret management via **External Secrets Operator**/**OpenBao**; least-privilege service accounts for controllers; supply-chain hygiene via **CodeQL**, **OpenSSF Scorecard** and organization-wide **Trivy** CVE scanning of dependencies and container images. - * *Loosening defaults:* adopters can relax tenant isolation (shared/flat networking), disable specific NetworkPolicies, or grant broader RBAC for single-tenant/lab use. `[maintainers: link docs on loosening tenant-isolation/network-policy defaults]` + * *Loosening defaults:* isolation between unrelated tenants is enforced by Cilium egress **by design**, and there is no supported mechanism to disable tenant isolation wholesale or to create flat/shared inter-tenant networking. What is documented is **opt-in relaxation to specific restricted services**: per the [tenants guide](https://cozystack.io/docs/v1.6/guides/tenants/), labels such as `policy.cozystack.io/allow-to-apiserver: "true"` and `policy.cozystack.io/allow-to-etcd: "true"` let tenant pods reach otherwise-restricted endpoints, and broader RBAC can be granted for single-tenant/lab use. (The earlier `isolated` tenant field was removed in favor of these opt-in policy labels.) * **Security Hygiene.** * Practices: GitHub review with required approvals; CI on every PR. The branch protection on `main` currently requires two status checks, `pre-commit` and `E2E Tests`. **CodeQL** static analysis runs on every PR, on push to `main` and weekly; new error-severity alerts are expected to be resolved or explicitly dismissed before merge, but CodeQL is **not yet a required check** — promoting it to one is an open item, and [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md) states it as an intention rather than a fact. **zizmor** audits the GitHub Actions workflows themselves and fails on findings such as unpinned actions or over-broad `permissions`; it runs two ways, as a hook inside the required `pre-commit` check (so it genuinely gates merges) and as a standalone workflow triggered only by changes under `.github/workflows/` or to `.github/zizmor.yml`, which is not itself a required check. **OpenSSF Scorecard** runs weekly and publishes to scorecard.dev; the OpenSSF Best Practices badge is maintained. Organization-wide **Trivy** scanning covers dependencies and container images on a 6-hour cycle for CRITICAL findings and weekly for the rest, with per-CVE triage tracked as issues (see Day 2 → Dependencies). **Renovate** automates dependency updates within the scope described in Day 2 → Dependencies. Component versions are pinned per release. - * Unmaintained-feature risk: components are curated and version-pinned per release; the Helm-packaged, swappable building-block design makes replacing a risky/unmaintainable component tractable. `[maintainers: describe the formal process for retiring/replacing a risky component]` + * Unmaintained-feature risk: components are curated and version-pinned per release; the Helm-packaged, swappable building-block design makes replacing a risky/unmaintainable component tractable. [GOVERNANCE.md](https://github.com/cozystack/cozystack/blob/main/GOVERNANCE.md) documents how bundled third-party components are sourced and held (vendored by digest under the upstream owner's namespace, not re-released by the project). A **formal governance workflow for deciding to retire or replace** a risky/unmaintained bundled component is **not yet documented**. * **Cloud Native Threat Modeling.** * *Least privilege:* controllers run with scoped RBAC; elevated privileges are limited to and documented for infrastructure components that genuinely need them (KubeVirt/CDI for virtualization, Piraeus/LINSTOR for block devices, Cilium/Kube-OVN for CNI). The published threat model is [`docs/security/threat-model.md`](https://github.com/cozystack/cozystack/blob/main/docs/security/threat-model.md). * *Certificate rotation:* automated via **cert-manager** (issuance/renewal/rotation) for platform TLS; Talos/Kubernetes-managed PKI for cluster components; tenant control-plane certs managed by **Kamaji**. @@ -81,25 +77,25 @@ ### Project Installation and Configuration -* As in Day 0 → Installation: Talos-based management cluster, then Cozystack via **Platform** or **BYOP** mode (OCI Helm installer + `cozystack` ConfigMap + MetalLB pool). All subsequent configuration is declarative (Kubernetes resources / Flux-managed HelmReleases); Day-2 changes are made by editing resources, not imperative steps. +* As in Day 0 → Installation: a management cluster (Talos on the recommended path; generic Linux/k3s or an existing Kubernetes cluster on the other paths), then Cozystack via **Platform** or **BYOP** mode (OCI Helm installer + `cozystack` ConfigMap + MetalLB pool). All subsequent configuration is declarative (Kubernetes resources / Flux-managed HelmReleases); Day-2 changes are made by editing resources, not imperative steps. ### Project Enablement and Rollback -* **Enable/disable in a live cluster.** Cozystack is the distribution rather than an add-on toggled inside a foreign cluster, so "enablement" = installing the operator bundle on the management cluster. Individual bundles/components and each tenant application are enabled/removed independently (create/delete the corresponding resource); removing a tenant application affects only that tenant, with no downtime to unrelated workloads. `[maintainers: confirm whether any core component change requires control-plane restart]` +* **Enable/disable in a live cluster.** On the Talos and generic-Linux paths Cozystack is the distribution that owns the cluster, so "enablement" = installing the operator bundle on the management cluster; on the bring-your-own-Kubernetes path it is installed onto an existing cluster instead. Individual bundles/components and each tenant application are enabled/removed independently (create/delete the corresponding resource); removing a tenant application affects only that tenant, with no downtime to unrelated workloads. `[maintainers: confirm whether any core component change requires control-plane restart]` * **How enablement changes default cluster behavior.** Installing Cozystack sets the CNI (Cilium/Kube-OVN), default StorageClass (LINSTOR), ingress, cert-manager, and the aggregated API as cluster defaults for subsequently created workloads. Cozystack expects to own the cluster, so a clean install does not disturb pre-existing unrelated workloads. -* **Testing enablement/disablement.** CI installs Cozystack end-to-end in ephemeral environments and provisions/destroys sample tenants and applications to validate create→ready→delete flows. Workflows: [`pull-requests.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/pull-requests.yaml) (per-PR checks), [`e2e-tag.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/e2e-tag.yaml) (tag/release e2e), [`nightly.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/nightly.yaml) (scheduled full run) and [`ui-test.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/ui-test.yaml) (dashboard). Test conventions are documented in [`docs/agents/e2e-testing.md`](https://github.com/cozystack/cozystack/blob/main/docs/agents/e2e-testing.md). **Current health of these workflows:** the gating checks are the per-PR ones — `pre-commit` and `E2E Tests` are the two required status checks on `main`, and a change cannot merge without them passing. The scheduled runs are another matter. `nightly.yaml` is **flapping rather than simply broken**, and has been for a month: the first scheduled run after the fix in [cozystack#3332](https://github.com/cozystack/cozystack/pull/3332) still failed, then it passed for three days, went red for five, passed for three more ending 2026-08-01, and has failed every scheduled run since 2026-08-02. Six of its last twenty-four scheduled runs passed. The current failure has **not been triaged to an issue** — several e2e failures from the same window are open and may share a cause, and the older nightly item in [cozystack#3278](https://github.com/cozystack/cozystack/issues/3278) describes the earlier never-completing failure rather than this one — so the honest statement is that the scheduled full-matrix run is unreliable and its current failure is not yet attributed. The tag-triggered `e2e-tag.yaml` also failed around the v1.6.1 release. Enablement and disablement coverage therefore rests on the per-PR e2e job today, and restoring the scheduled full-matrix run is outstanding work. +* **Testing enablement/disablement.** CI installs Cozystack end-to-end in ephemeral environments and provisions/destroys sample tenants and applications to validate create→ready→delete flows. Workflows: [`pull-requests.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/pull-requests.yaml) (per-PR checks), [`e2e-tag.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/e2e-tag.yaml) (tag/release e2e), [`nightly.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/nightly.yaml) (scheduled full run) and [`ui-test.yaml`](https://github.com/cozystack/cozystack/blob/main/.github/workflows/ui-test.yaml) (dashboard). Test conventions are documented in [`docs/agents/e2e-testing.md`](https://github.com/cozystack/cozystack/blob/main/docs/agents/e2e-testing.md). **Current health of these workflows:** the gating checks are the per-PR ones — `pre-commit` and `E2E Tests` are the two required status checks on `main`, and a change cannot merge without them passing. The scheduled runs are another matter. `nightly.yaml` has been **unreliable on the scheduled full run** for a month: the first scheduled run after the fix in [cozystack#3332](https://github.com/cozystack/cozystack/pull/3332) still failed, then it passed for three days, went red for five, passed for three more ending 2026-08-01, and has failed nearly every scheduled run since, with only isolated passes on 2026-08-17 and 2026-09-03. As of 2026-09-17, 1 of the last 24 scheduled runs passed (on 2026-09-03), and every scheduled run after it has failed. The current failure has **not been triaged to an issue** — several e2e failures from the same window are open and may share a cause, and the older nightly item in [cozystack#3278](https://github.com/cozystack/cozystack/issues/3278) describes the earlier never-completing failure rather than this one — so the honest statement is that the scheduled full-matrix run is unreliable and its current failure is not yet attributed. The tag-triggered `e2e-tag.yaml` also failed around the v1.6.1 release. Enablement and disablement coverage therefore rests on the per-PR e2e job today, and restoring the scheduled full-matrix run is outstanding work. * **Resource cleanup incl. CRDs.** Deleting an application resource triggers Flux/Helm to remove its HelmRelease and owned objects; finalizers ensure dependent resources (PVCs, VM disks, LB IPs) are released. `[maintainers: confirm CRD-cleanup behavior on full uninstall and any manual steps]` ### Rollout, Upgrade and Rollback Planning * **Compatibility with Kubernetes & frequency.** Cozystack ships a curated, tested Kubernetes version per release; tenant clusters are Certified Kubernetes following upstream skew/support. Platform upgrades are delivered as new Cozystack releases. The authoritative list of what has shipped is the [GitHub Releases page](https://github.com/cozystack/cozystack/releases); which release lines are still receiving security fixes is defined as a rolling rule in [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md): the latest stable minor and the two preceding minors are supported, older `1.x` minors receive critical and upgrade-blocking fixes at maintainer discretion, `v0.41.x` is end of life and anything earlier is unsupported. Specific version numbers are deliberately not restated here, because they drift with every release. Note that being in a supported line does not entail receiving every fix: the project has no documented rule for which supported lines a given fix is backported to, and that is recorded in `SECURITY.md` rather than presented as a policy. * **Upgrade procedure** ([upgrade docs](https://cozystack.io/docs/v1.6/operations/cluster/upgrade/)): run the troubleshooting/health checklist; annotate the `cozy-system` namespace and `cozystack-version` ConfigMap with `helm.sh/resource-policy=keep`; `helm upgrade cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer --version X.Y.Z` (**do not** use `--reuse-values` — the chart pins OCI repo/package versions); verify pods and `kubectl get hr -A`. -* **Rollback.** GitOps/Helm-managed with versioned releases; roll back by targeting the previous release (previous HelmRelease revisions) via the same mechanism. Stateful data (databases, VM disks) persists on replicated storage across control-plane rollbacks. `[maintainers: confirm a documented rollback runbook / version-skip constraints]` -* **How a rollout/rollback can fail & impact.** Risks: incompatible CRD schema change, storage/CNI component upgrades requiring Talos node reboots, or a database-operator major bump. Impact is scoped to the component; running tenant workloads generally continue, though storage/CNI upgrades can transiently affect networking or volume attachment during node rolls. `[maintainers: confirm whether upgrades trigger Talos node reboots]` +* **Rollback.** GitOps/Helm-managed with versioned releases; roll back by targeting the previous release (previous HelmRelease revisions) via the same mechanism. Stateful data (databases, VM disks) persists on replicated storage across control-plane rollbacks. **No dedicated rollback runbook is published**, and version-skip constraints for downgrades are not documented; the [upgrade docs](https://cozystack.io/docs/v1.6/operations/cluster/upgrade/) currently cover the forward path only. +* **How a rollout/rollback can fail & impact.** Risks: incompatible CRD schema change, storage/CNI component upgrades requiring node reboots (on the Talos path), or a database-operator major bump. Impact is scoped to the component; running tenant workloads generally continue, though storage/CNI upgrades can transiently affect networking or volume attachment during node rolls. `[maintainers: confirm whether upgrades trigger Talos node reboots]` * **Metrics that should inform a rollback.** Control-plane/API availability & latency, Flux reconciliation failures (`kubectl get hr -A | grep -v True`), DRBD/LINSTOR replication health (`linstor resource list --faulty`), CNI/pod-networking errors, tenant application readiness. * **Upgrade/rollback testing (upgrade→downgrade→upgrade).** `[maintainers: describe the CI matrix testing N-1→N upgrades and the downgrade path; link results]` -* **Deprecations/removals communication.** Release notes/CHANGELOG, GitHub release announcements, docs and community channels; API deprecations follow Kubernetes-style windows. `[maintainers: link the deprecation policy]` -* **Alpha/beta capabilities in a rollout.** Experimental bundles/apps are opt-in and marked as such; only Stable/Patch releases are recommended for production. `[maintainers: confirm how alpha/beta features are flagged per application]` +* **Deprecations/removals communication.** Release notes/CHANGELOG ([`docs/changelogs/`](https://github.com/cozystack/cozystack/tree/main/docs/changelogs)), GitHub release announcements, docs and community channels. A dedicated, documented deprecation policy/window is **not yet published** — it is folded into the planned API Stability Policy (see Day 0 → API Design → Versioning). +* **Alpha/beta capabilities in a rollout.** Only Stable/Patch releases are recommended for production; experimental bundles/apps are opt-in. **There is no formal per-application maturity field** (e.g. alpha/beta/stable) in [ApplicationDefinition](https://cozystack.io/docs/v1.6/cozystack-api/application-definitions/) today; maturity is expressed informally in docs and release notes. Adding maturity/compatibility metadata is a roadmap (Marketplace) item. --- @@ -107,8 +103,8 @@ ### Scalability/Reliability -* **Increasing size/count of API objects.** Scale horizontally by adding Talos nodes and by creating more tenants/applications (each an independent set of Flux-reconciled Kubernetes objects); storage scales by adding disks/nodes to the LINSTOR pool; **Cluster Autoscaler** and **Vertical Pod Autoscaler** assist node/pod right-sizing. -* **SLOs/SLIs.** `[maintainers: state whether the project publishes reference SLOs/SLIs for the management API, tenant control-plane availability and storage, or note that SLOs are adopter-defined and list the SLIs exposed]` +* **Increasing size/count of API objects.** Scale horizontally by adding nodes (Talos or generic-Linux, per the install path) and by creating more tenants/applications (each an independent set of Flux-reconciled Kubernetes objects); storage scales by adding disks/nodes to the LINSTOR pool; **Cluster Autoscaler** and **Vertical Pod Autoscaler** assist node/pod right-sizing. +* **SLOs/SLIs.** The project does **not publish prescriptive reference SLOs**; availability and latency targets are **adopter-defined**, because they depend on the operator's hardware tier, storage topology and tenant density. Cozystack exposes the **SLIs** needed to define and measure them: management-API availability and latency, Flux reconciliation success/failure per HelmRelease, tenant control-plane (Kamaji) pod readiness, DRBD/LINSTOR replication health, storage I/O, CNI/pod-networking errors, and ingress/LB health — all surfaced through VictoriaMetrics/Grafana (see Observability). * **Operations increasing in time under existing SLIs.** Provisioning large tenant clusters, VM live-migration, and DRBD full-resync after node replacement scale with size/data volume. `[maintainers: confirm]` * **Resource-usage increase from enabling the project.** Platform components (Cilium, Kube-OVN, KubeVirt/CDI, LINSTOR, VictoriaMetrics/VictoriaLogs, Flux, DBaaS operators) consume a baseline on the management cluster; VM/DB workloads dominate at scale. The hardware tiers above bound expected footprints. `[maintainers: provide measured baseline overhead of platform components]` * **Conditions causing node-resource exhaustion.** Many VMs/pods per node (PIDs, sockets, hugepages), inodes on storage nodes, DRBD/LINSTOR memory under heavy replication, conntrack under high connection counts. Quotas/limits, VPA and MetalLB/ingress sizing mitigate. `[maintainers: confirm]` @@ -121,7 +117,7 @@ * **Signals produced** ([monitoring docs](https://cozystack.io/docs/v1.6/operations/services/monitoring/)): metrics via **VictoriaMetrics (VMCluster)** collected by **VMAgent** (Prometheus-compatible); logs via **VictoriaLogs (VLogs)** collected by **Fluent Bit** (DaemonSet); dashboards via **Grafana**; alerting via **Alerta**; plus **kube-state-metrics** and **node-exporter**. Kubernetes events and component logs in standard formats. * **Audit logging.** Kubernetes API audit logging can be enabled on management and tenant control planes through the standard kube-apiserver audit policy. **No default audit policy is shipped** by Cozystack — no audit-policy manifest exists in any package — so audit logging is opt-in and the policy is authored by the adopter. Enabling it by default, with a documented reference policy, is an open improvement rather than an existing feature. * **Dashboards.** Pre-built Grafana dashboards for cluster overview, node metrics, storage, KubeVirt/VMs and per-application health, plus user-created dashboards (Grafana Operator manages dashboards as CRDs). -* **Cost/FinOps surfacing.** Per-tenant/per-application resource usage is observable via metrics and Kubernetes resource accounting (chargeback/showback), integrating with provider billing/portal. `[maintainers: link any FinOps/billing integration]` +* **Cost/FinOps surfacing.** Per-tenant/per-application resource usage is observable via metrics and Kubernetes resource accounting (a chargeback/showback basis). **No billing/FinOps integration is implemented today**; a Tenant FinOps API (OpenCost-compatible) is a roadmap item. Adopters integrate the exposed usage metrics with their own provider billing/portal. * **Health parameters.** Control-plane/API availability, Flux reconciliation status, Talos/node health, DRBD replication state, pod/VM readiness, ingress/LB health. * **Determining the project is in use.** Presence of Cozystack CRDs/aggregated APIs (`apps.cozystack.io`) and reconciled platform HelmReleases; existence of `Tenant`/application resources. * **Knowing it works for your instance.** Dashboards green; all HelmReleases `Ready`; sample tenant/app provisions successfully; storage replication healthy. @@ -134,12 +130,12 @@ * **Source Composition Analysis (SCA).** Four layers. (1) **Static analysis of first-party code** — **CodeQL** on every PR, on push to `main` and weekly, reporting only alerts new relative to `main`; expected to be resolved or dismissed before merge, though not yet a required check. (2) **Static analysis of the CI supply chain itself** — **zizmor** audits the GitHub Actions workflow definitions for unpinned actions, over-broad `permissions` and similar findings; it gates merges through the required `pre-commit` check, with an additional workflow covering tree-wide runs when the workflow definitions themselves change. (3) **Supply-chain posture** — **OpenSSF Scorecard** weekly, published on scorecard.dev, plus the OpenSSF Best Practices badge. (4) **Dependency and container-image CVE scanning** — an organization-wide pipeline built on **[Trivy](https://trivy.dev/)** running against every non-fork, non-archived repository in the `cozystack` organization. * **How the CVE-scanning pipeline works.** Discovery walks each repository for Go modules (`go.mod`, direct dependencies), base images from Dockerfile `FROM` directives, and container images referenced in `values.yaml` under `packages/apps/*`, `packages/system/*` and `packages/core/*`. Trivy then scans both the repository filesystem — which is where transitive Go dependencies and lock files are actually covered, since discovery itself reads only direct requirements — and every resolved container image, against NVD, vendor advisories and the GitHub Advisory Database. **A known limitation:** image references assembled at render time by the `cozy-lib.image` helper are not literal strings in `values.yaml` and are therefore invisible to values-file discovery, the same blind spot that led to disabling Renovate's `helm-values` manager. Those images are covered only insofar as they appear elsewhere; closing the gap requires resolving refs through the rendered chart rather than the values file. CRITICAL findings are scanned and reported every 6 hours; HIGH/MEDIUM/LOW are collected into a weekly report. Four filters run **before** an issue is created: findings with no upstream fix older than 365 days; dev and build-only dependencies (`golang:`/`node:`/`python:` images, `core/testing`, `hack/`, `e2e/`); CVEs already carrying a maintainer triage decision; and findings with no severity assigned in any database. As a result, every new **actionable** CVE becomes a tracked issue with severity labels, affected components, fix version and a triage checklist, while roughly 20–30% of raw output is suppressed as noise. Triage decisions are persisted, so a settled finding is not re-reported. * **Public verifiability of the pipeline — an open problem.** The pipeline, its reports and its triage state live in `cozystack/security-scanner`, which returns 404 to anyone outside the organization, so **none of the above is currently checkable by a reviewer without privileged access.** The line that needs drawing is not per-repository but per-artifact, and it falls in a specific place: **severity-level aggregate counts are publishable; naming an individual finding that is not yet fixed is not**, because a CVE ID plus an affected package plus this project's public, version-pinned component list is enough to locate unpatched exposure in a running deployment. Against that test, the pipeline policy (`SECURITY_PIPELINE_POLICY.md`), the severity and SLA matrix (`SECURITY_TRIAGE_MATRIX.md`), the finding status model (`SECURITY_FINDING_STATUS_MODEL.md`) and the scanning scripts hold no finding data and are publishable as they stand — and they are the part that actually evidences the control. Monthly reports for every month the pipeline has run are now published in-tree at [`docs/security/reports/`](https://github.com/cozystack/cozystack/tree/main/docs/security/reports), with identifiers withheld unless the fix is merged and released and a generator guard that refuses to emit a file containing an unreleased one. They are accompanied by a [backlog reconciliation](https://github.com/cozystack/cozystack/blob/main/docs/security/reports/reconciliation-2026-08.md) auditing every tracked finding against what the project actually shipped. That audit revised the disclosure rule in two directions that make it more open: findings with no upstream fix will be named with mitigation guidance, since there is nothing to wait for and an adopter needs to know when a mitigation is the only available control; and remediation is no longer attributed to individual months, because a count of still-open findings crossed with one month's publicly enumerable component bumps narrows the search for unpatched exposure more than a bare count does. What remains is publishing the pipeline's own policy documents and scanning scripts, which hold no finding data. `[maintainers: publish SECURITY_PIPELINE_POLICY.md, SECURITY_TRIAGE_MATRIX.md, SECURITY_FINDING_STATUS_MODEL.md and the scanning scripts, so the method is auditable independently of the numbers]` -* **Acting on SCA & timescale.** Two tracks with distinct **targets**. **Reported vulnerabilities** (Cozystack-specific code, packaging, defaults, integration logic) follow [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md): acknowledgement ≤3 business days, initial triage ≤7 business days, remediation targets Critical ≈14 days, High ≈30 days, Medium ≈90 days, Low next release, under a coordinated-disclosure window of up to 90 days. **Scanner findings in bundled upstream components** are a separate track with a tighter triage clock: Critical triaged within 1 business day against a 7-day fix target, High within 3 business days / 30 days, Medium within 10 business days / 90 days, Low within 30 days best-effort. Triage follows an explicit decision tree (is the CVE in a shipped artifact rather than dev/test tooling; is the vulnerable path reachable in Cozystack's usage; has the distribution already backported the fix; is an upstream fix available) and resolves to one of `confirmed`, `false-positive`, `accepted-risk`, `in-progress` or `fixed`. Confirmed findings ship as a pinned-component version bump in the next Cozystack release, or as a patch release on a supported line when severity warrants it. **Attainment is uneven, and an audit of the pipeline's own output found its state misleading in both directions.** Detection keeps up. **Triage does not**: the backlog was decided in a single bulk pass rather than continuously, and the rotating role that owns the triage clock is defined but unassigned — that is the genuine unmet target. **Remediation, however, is happening and the tracker cannot see it**, because nothing writes back from the main repository and so no finding has ever been marked fixed. Reconciling every tracked finding against merged changes, release tags and the pipeline's own weekly scan reports establishes two things. First, a clear majority of confirmed findings relate to a component whose in-tree version now carries the fix, concentrated in a single remediation wave in June 2026 rather than a steady stream. Second, four component bumps are available upstream and not yet taken. The same audit found that **57% of the backlog was never applicable or cannot be assessed** — the largest block being findings against packages that contain no executable vulnerable code, chiefly kernel headers, since a container never provides a running kernel. Reachable exposure in a stock installation is estimated at 2–5% of the tracker. Seven defects were found in the pipeline itself, including deduplication by CVE identifier alone and a broken image-attribution step that left 16% of findings unattributable. Figures, method and defect list: [backlog reconciliation](https://github.com/cozystack/cozystack/blob/main/docs/security/reports/reconciliation-2026-08.md). `[maintainers: fill the Security Champion role, close the write-back loop, and take the four bumps that are available upstream and not yet pulled in]` +* **Acting on SCA & timescale.** Two tracks with distinct **targets**. **Reported vulnerabilities** (Cozystack-specific code, packaging, defaults, integration logic) follow [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md): acknowledgement ≤3 business days, initial triage ≤7 business days, remediation targets Critical ≈14 days, High ≈30 days, Medium ≈90 days, Low next release, under a coordinated-disclosure window of up to 90 days. **Scanner findings in bundled upstream components** are a separate track with a tighter triage clock: Critical triaged within 1 business day against a 7-day fix target, High within 3 business days / 30 days, Medium within 10 business days / 90 days, Low within 30 days best-effort. Triage follows an explicit decision tree (is the CVE in a shipped artifact rather than dev/test tooling; is the vulnerable path reachable in Cozystack's usage; has the distribution already backported the fix; is an upstream fix available) and resolves to one of `confirmed`, `false-positive`, `accepted-risk`, `in-progress` or `fixed`. Confirmed findings ship as a pinned-component version bump in the next Cozystack release, or as a patch release on a supported line when severity warrants it. **Attainment is uneven, and an audit of the pipeline's own output found its state misleading in both directions.** Detection keeps up. **Triage now has an owner, and continuous attainment is what is left to demonstrate.** The backlog was decided in a single bulk pass rather than continuously, because the rotating role that owns the triage clock, though defined, was unassigned; that role has now been assigned to @tym83, who runs a weekly triage pass, so sustained continuous attainment of the triage SLAs is the remaining target rather than the vacancy itself. **Remediation, however, is happening and the tracker cannot see it**, because nothing writes back from the main repository and so no finding has ever been marked fixed. Reconciling every tracked finding against merged changes, release tags and the pipeline's own weekly scan reports establishes two things. First, a clear majority of confirmed findings relate to a component whose in-tree version now carries the fix, concentrated in a single remediation wave in June 2026 rather than a steady stream. Second, four component bumps are available upstream and not yet taken. The same audit found that **57% of the backlog was never applicable or cannot be assessed** — the largest block being findings against packages that contain no executable vulnerable code, chiefly kernel headers, since a container never provides a running kernel. Reachable exposure in a stock installation is estimated at 2–5% of the tracker. Seven defects were found in the pipeline itself, including deduplication by CVE identifier alone and a broken image-attribution step that left 16% of findings unattributable. Figures, method and defect list: [backlog reconciliation](https://github.com/cozystack/cozystack/blob/main/docs/security/reports/reconciliation-2026-08.md). `[maintainers: close the write-back loop and take the four bumps that are available upstream and not yet pulled in]` ### Troubleshooting -* **Recovery when a key component fails** ([troubleshooting](https://cozystack.io/docs/v1.6/operations/troubleshooting/)): Kubernetes/etcd HA and Talos self-healing recover control-plane node loss; **Kamaji** reschedules tenant control planes; **DRBD/LINSTOR** promotes a healthy replica; **Flux** re-reconciles drifted/removed platform objects; databases recover via their operators; **Velero** provides backup/restore. Diagnostics: `kubectl get node`; `kubectl get hr -A | grep -v True`; `kubectl get pod -A | grep -v 'Running\|Completed'`; `linstor node/storage-pool/resource list [--faulty]`; operator logs `kubectl logs -n cozy-system deploy/cozystack-operator`; dedicated guides for [etcd](https://cozystack.io/docs/v1.6/operations/troubleshooting/etcd/), [Flux CD](https://cozystack.io/docs/v1.6/operations/troubleshooting/flux-cd/) and [Kube-OVN](https://cozystack.io/docs/v1.6/operations/troubleshooting/kube-ovn/). -* **Known failure modes.** Documented areas include broken HelmReleases (`DependenciesNotReady` when a required package is disabled/missing), LINSTOR/DRBD faulty resources, Kube-OVN OVN-database corruption (recover via `ovn-appctl` procedures), and etcd quorum issues — each with a dedicated troubleshooting guide. `[maintainers: consolidate a "known failure modes + mitigations" list if not already published]` +* **Recovery when a key component fails** ([troubleshooting](https://cozystack.io/docs/v1.6/operations/troubleshooting/)): Kubernetes/etcd HA (and Talos self-healing on the Talos path) recover control-plane node loss; **Kamaji** reschedules tenant control planes; **DRBD/LINSTOR** promotes a healthy replica; **Flux** re-reconciles drifted/removed platform objects; databases recover via their operators; **Velero** provides backup/restore. Diagnostics: `kubectl get node`; `kubectl get hr -A | grep -v True`; `kubectl get pod -A | grep -v 'Running\|Completed'`; `linstor node/storage-pool/resource list [--faulty]`; operator logs `kubectl logs -n cozy-system deploy/cozystack-operator`; dedicated guides for [etcd](https://cozystack.io/docs/v1.6/operations/troubleshooting/etcd/), [Flux CD](https://cozystack.io/docs/v1.6/operations/troubleshooting/flux-cd/) and [Kube-OVN](https://cozystack.io/docs/v1.6/operations/troubleshooting/kube-ovn/). +* **Known failure modes.** Documented areas include broken HelmReleases, LINSTOR/DRBD faulty resources, Kube-OVN OVN-database recovery, and etcd quorum issues, each with a dedicated guide under the [troubleshooting section](https://cozystack.io/docs/v1.6/operations/troubleshooting/) ([etcd](https://cozystack.io/docs/v1.6/operations/troubleshooting/etcd/), [Flux CD](https://cozystack.io/docs/v1.6/operations/troubleshooting/flux-cd/), [Kube-OVN](https://cozystack.io/docs/v1.6/operations/troubleshooting/kube-ovn/), [LINSTOR controller](https://cozystack.io/docs/v1.6/operations/troubleshooting/linstor-controller/) / [database](https://cozystack.io/docs/v1.6/operations/troubleshooting/linstor-database/), [monitoring](https://cozystack.io/docs/v1.6/operations/troubleshooting/monitoring-troubleshooting/)). These are per-topic pages; a **single consolidated "known failure modes + mitigations" index is not yet published.** ### Compliance @@ -147,12 +143,12 @@ * **Alignment with CNCF attribution recommendations.** * *Notices for third-party code in source files:* retained in-file / in `LICENSE`/`NOTICE`. `[maintainers: confirm]` * *Notices for unmodified third-party components in the repo:* retained alongside the pinned component. `[maintainers: confirm]` - * *Notices in distributed build artifacts (images/binaries):* `[maintainers: describe how license notices/SBOM are included in container images]` + * *Notices in distributed build artifacts (images/binaries):* SBOM generation is implemented but **off by default** — `SBOM=1` in [`hack/common-envs.mk`](https://github.com/cozystack/cozystack/blob/main/hack/common-envs.mk) attaches a CycloneDX SBOM as a buildx attestation, disabled pending registry-referrer verification (tracked in [cozystack#2937](https://github.com/cozystack/cozystack/issues/2937)); release images are not yet signed and build provenance is disabled (see Day 0 → Security → Supply-chain gaps). Bundled-component license terms are catalogued in the [licenses documentation](https://cozystack.io/docs/v1.6/operations/configuration/licenses/). ### Security * **Access control execution.** Kubernetes RBAC end-to-end; per-tenant namespaces, RoleBindings, quotas and NetworkPolicies; isolated tenant API servers (Kamaji + dedicated etcd); Keycloak OIDC/SAML for federated auth; secrets via External Secrets Operator / OpenBao. -* **Security reporting team & rotation.** Reporting via **GitHub Private Vulnerability Reporting** ([advisories/new](https://github.com/cozystack/cozystack/security/advisories/new)) or a private maintainer contact; any maintainer can receive and route a report. Current security response team: **@kvaps, @lexfrei, @tym83, @matthieu-robin, @mattia-eleuteri**. `[maintainers: describe how security-team membership reflects community/organizational diversity and how members are invited/rotated — this rotation policy is not yet documented in SECURITY.md]` +* **Security reporting team & rotation.** Reporting via **GitHub Private Vulnerability Reporting** ([advisories/new](https://github.com/cozystack/cozystack/security/advisories/new)) or a private maintainer contact; any maintainer can receive and route a report. Current security response team: **@kvaps, @lexfrei, @tym83, @matthieu-robin, @mattia-eleuteri** — spanning more than one organization (Ænix, HIDORA) and more than one country, rather than a single employer. A rotating **Security Champion** role (a fixed term with a handoff overlap) owns the CVE-triage clock; the role is **currently held by @tym83**, who runs a weekly triage pass over scanner findings, while technical reachability of a given CVE in bundled components is assessed together with the security team. This rotation cadence and the team's diversity statement are published in the project's [`SECURITY.md`](https://github.com/cozystack/cozystack/blob/main/SECURITY.md). ---