cozystack/GENERAL_TECHNICAL_REVIEW.md

General Technical Review - Cozystack / Incubation

• Project: Cozystack
• Project Version: v1.x (current stable line). The authoritative list of releases is the GitHub Releases page; which release lines receive security fixes is defined in SECURITY.md. [maintainers: pin the exact release tag being submitted]
• Website: https://cozystack.io
• Date Updated: 2026-08-10
• Template Version: v1.0
• Description: Cozystack is a free, Apache-2.0 open-source platform and framework for building clouds. It turns a set of bare-metal servers into a system with a native Kubernetes REST API for spawning Kubernetes clusters, Databases-as-a-Service, virtual machines, load balancers, HTTP-caching and other services. It is a CNCF Sandbox project and a Certified Kubernetes distribution.

Status: DRAFT — work in progress. This questionnaire is being actively completed and reviewed. Answers that repository evidence settles have been written and verified against the tree; the remaining [maintainers: …] markers are of two kinds, and only the first kind blocks submission. Some editorial markers are certainly still closable from the tree — reviewers should treat an open marker as "not yet checked" rather than "checked and unanswerable".

Substantive gaps — need measurement, publication or a decision: (1) load/scale test results and tested maximums (nodes, tenants, VMs, PVs) with methodology; (2) whether the project publishes reference SLOs/SLIs or treats them as adopter-defined; (3) measured baseline resource overhead of the platform components; (4) the upgrade→downgrade→upgrade CI matrix and its results; (5) whether platform upgrades trigger Talos node reboots; (6) the security-response-team rotation and diversity policy — a rotating Security Champion role with a 6-month term and a handoff overlap is defined in the CVE-scanning pipeline's own governance document, but that document is unpublished, the current holder of the role is unassigned, and nothing about rotation or diversity appears in the public SECURITY.md; (7) publication of the CVE-scanning pipeline's policy documents and aggregate reports, without which the SCA answer below is not verifiable by a reviewer outside the organization.

Editorial markers — the rest are requests to link an existing document or confirm a detail, and are resolved as the corresponding docs are located or written.

Notes for reviewers: answers cover Day 0 and Day 1 (required for Incubation, cumulative) plus Day 2 where it already applies. Doc links are pinned to the versioned /docs/v1.6/ tree; repo links are on main and should be pinned to the submitted tag before final submission. Version numbers are deliberately not restated in prose where a canonical source exists — the Releases page and SECURITY.md are cited instead, so the document does not drift with each release.

---

Day 0 - Planning Phase

Scope

• Roadmap / scope / maintainer ladder. Scope for mid-to-long-term features is managed in the open on the public GitHub Project board, GitHub Issues/Discussions, the CNCF Slack #cozystack channel, Telegram, and the regular Cozystack community meetings. Roles and decision-making are defined in GOVERNANCE.md: four roles — Users, Contributors, Directors, Maintainers — decide by consensus, falling back to a simple-majority vote (one vote per Maintainer and Director). The promotion path is defined in CONTRIBUTOR_LADDER.md: Community Participant → Contributor → Reviewer → Maintainer, with concrete criteria (Reviewer: ≥15 PRs/yr, ≥20 reviews, ≥6 months, two sponsors from different employers; Maintainer: ≥6 months as Reviewer, ≥10 h/month), promotions are done by pull request against the roster files in the repository root and .github/: .github/CODEOWNERS records the area-scoped Reviewer role and gates merges on the matching paths, and MAINTAINERS.md carries three sections — Active Maintainers, Reviewers and Emeritus Maintainers — naming people, while CODEOWNERS is the authoritative path-to-Reviewer mapping. The two sets are deliberately different: a Reviewer owns a code area, a Maintainer is responsible for the whole project.
• Target personas. (1) Platform/infrastructure engineers and SRE/DevOps teams building private or public clouds; (2) service providers / hosting companies / data centers offering IaaS, DBaaS and Managed-Kubernetes; (3) enterprises consolidating workloads on bare metal and seeking a self-hosted public-cloud alternative or a VMware replacement.
• Primary use case. Turn bare-metal servers into a multi-tenant cloud exposing a Kubernetes-native REST API for self-service provisioning of tenant Kubernetes clusters, VMs, managed databases, load balancers and caching.
• Additional use cases. Private/hybrid cloud; managed-services platform for providers; dev/test environments; AI-ready GPU infrastructure; on-prem VMware/vSphere replacement.
• Explicitly unsupported. Cozystack is a full distribution that owns the cluster and the nodes (Talos-based) — it is not an add-on installed into an existing, externally-managed production Kubernetes cluster, is not a public-cloud managed-Kubernetes add-on, requires a minimum of 3 servers, and targets x86_64 (no Windows nodes). Single-node/2-node production is discouraged.
• Organizations that benefit. Hosting/cloud providers, data centers, telcos, financial services, and any platform-engineering organization needing sovereign, self-hosted cloud infrastructure.
• End-user research. Public adopters are listed in ADOPTERS.md. [maintainers: link any published case studies / end-user survey results]

Usability

• How personas interact. Administrators install and operate the management cluster; tenants/users interact declaratively via the Kubernetes API (kubectl apply of Cozystack application resources such as Kubernetes, VMInstance, Postgres, Bucket, Tenant, …), via the web dashboard, or via any Kubernetes-compatible tooling/CI, because every object is a native Kubernetes resource.
• UX/UI. Primary interface is the Kubernetes API surfaced through Cozystack's aggregated API layer (see API Design), so standard Kubernetes UX applies (kubectl, GitOps, client libraries). A web dashboard provides self-service creation/monitoring of applications and tenants, and shipped Grafana dashboards provide observability.
• Integration in production. Applications are Kubernetes CRs backed by Helm releases reconciled by Flux CD, so Cozystack integrates natively with GitOps pipelines, CI/CD and any operator ecosystem. Tenant Kubernetes clusters are standard Certified Kubernetes clusters usable with the entire CNCF landscape. Federated auth via Keycloak (OIDC/SAML) — see OIDC docs.

Design

• Design principles / best practices. Everything-is-a-Kubernetes-resource (infra, services and apps modeled uniformly as CRs and reconciled by controllers via GitOps); immutable infrastructure (Talos Linux, no SSH/no shell on nodes); composition of established CNCF/upstream building blocks rather than re-implementation; secure-by-default multi-tenancy (namespace/Tenant isolation, network policies, isolated tenant API servers); declarative, versioned Helm-based packaging.
• Architecture requirements & environments. A management cluster (the platform) provisions tenant clusters (Kubernetes-in-Kubernetes: Kamaji runs tenant control planes as pods, Cluster API with Kamaji + KubeVirt providers manages lifecycle, worker nodes/VMs on KubeVirt). Hardware tiers (hardware-requirements): Minimal/PoC 3 nodes × 8 cores / 24 GB / 50 GB + 256 GB SSD / 1 Gbps (1–2 tenants, ≤3 clusters); Small production 3 nodes × 16–32 cores / 64 GB / 100 GB + 1–2 TB (5–10 tenants); Large production 6+ nodes × 32–64 cores / 128–256 GB / 200 GB + 4–10 TB NVMe / 10 Gbps (20+ tenants). Low latency between nodes is required; VM-based nodes need nested virtualization and CPU model host.
• In-cluster service dependencies. Full platform stack (platform-stack): Talos Linux, Kubernetes; networking Cilium (eBPF CNI + NetworkPolicy), Kube-OVN (VPC isolation, floating IPs, tenant segmentation), MetalLB, ingress-nginx, Multus CNI, CoreDNS, ExternalDNS, Kilo (WireGuard mesh); virtualization KubeVirt + CDI; storage LINSTOR/DRBD via Piraeus, SeaweedFS (object/Bucket), Velero (backup/restore), CSI Driver NFS; GitOps/automation Flux, Aenix etcd Operator (dedicated etcd for tenant control planes), cert-manager, External Secrets Operator; observability VictoriaMetrics + VictoriaLogs, Grafana, Alerta, VMAgent, Fluent Bit, kube-state-metrics, node-exporter; autoscaling Vertical Pod Autoscaler, Cluster Autoscaler; databases PostgreSQL, MariaDB, MongoDB (Percona), ClickHouse, OpenSearch, Qdrant, FoundationDB, Redis; messaging Kafka (Strimzi), NATS, RabbitMQ; identity/registry Keycloak, Harbor, OpenBao.
• Identity & Access Management. Kubernetes-native RBAC end-to-end; multi-tenancy via the Tenant resource with per-tenant namespaces, scoped RoleBindings, NetworkPolicies and quotas; each tenant Kubernetes cluster has a fully isolated API server (separate Kamaji control plane + dedicated etcd). Federated SSO via Keycloak (OIDC/SAML). Secrets integration via External Secrets Operator and OpenBao (PKI/dynamic secrets).
• Sovereignty. Fully self-hosted on the operator's own hardware; no dependency on any public cloud or external SaaS control plane — all data and control planes remain on-premises. This is a core design goal.
• Compliance. Apache-2.0; ships a Security Self-Assessment (see Security) and OpenSSF Best Practices + Scorecard. Cozystack provides isolation/RBAC/audit/network-policy/PKI primitives adopters use to build compliant platforms rather than claiming formal certifications itself. [maintainers: list any published compliance mappings]
• High Availability. Management control plane HA (≥3 nodes, dedicated etcd); tenant control planes (Kamaji) run as replicated pods and reschedule on failure; storage synchronously replicated (DRBD, 2–3 replicas); MetalLB failover for LB IPs; ingress replicated.
• Resource requirements (CPU/Network/Memory). See hardware tiers above — from 3 × (8 cores / 24 GB) minimal to 6+ × (32–64 cores / 128–256 GB) at scale; 1 Gbps minimum, 10 Gbps recommended for production.
• Storage requirements. Persistent replicated block storage (LINSTOR/DRBD via Piraeus) for stateful workloads (DB and VM disks) and object storage (SeaweedFS, S3-compatible) for the Bucket service/backups; Velero for backup/restore; CSI Driver NFS for NFS-backed volumes. Nodes need a dedicated raw secondary disk for LINSTOR (single-disk is possible but discouraged).
• API Design:
  • Topology & conventions. Cozystack exposes applications through a Kubernetes API-aggregation layer: a dynamically-generated aggregated API server surfaces each Cozystack "application" as a first-class, namespaced Kubernetes resource under Cozystack API groups (apps.cozystack.io), each backed by an ApplicationDefinition and a Helm chart/HelmRelease reconciled by Flux. Conventions follow Kubernetes API machinery (group/version/kind, spec/status). Reference: REST API, Go Types, ApplicationDefinition; background: blog "How we built a dynamic Kubernetes API server for the API aggregation layer in Cozystack".
  • Defaults. Secure/production-oriented defaults: RBAC on, per-tenant isolation, network policies, replicated storage, sane per-app sizing overridable via the resource spec.
  • Additional configuration. Cluster-wide settings via the cozystack ConfigMap (bundle/variant, pod/service CIDRs, root host, API endpoint, storage) plus a MetalLB address pool; per-app tuning via the resource spec.
  • New/changed API types & cloud-provider calls. Introduces its own CRDs/aggregated types (apps.cozystack.io, Tenant, per-service kinds). Makes no external cloud-provider API calls — everything is on-prem.
  • Compatibility with the Kubernetes API server. Integrates via standard Kubernetes API-aggregation (APIService); does not fork/modify core kube-apiserver; tenant clusters run upstream Certified Kubernetes.
  • Versioning & breaking changes. API groups versioned per Kubernetes conventions; breaking changes gated via release notes/deprecation windows. [maintainers: confirm and link the API stability/deprecation policy]
• Release process. Semantic Versioning; releases on GitHub (stable vX.Y.0, patch vX.Y.Z). Artifacts are container images and Helm charts distributed via OCI (ghcr.io/cozystack/cozystack). Component versions are pinned per Cozystack release so an upgrade is a single coordinated bump. Only Stable/Patch releases are recommended in production (avoid alpha/beta/rc).

Installation

• How it is installed/initialized. (1) Provision bare-metal (or nested-virt) nodes with Talos Linux (PXE/ISO) and bootstrap a management Kubernetes cluster (Talos install, Kubernetes install). (2) Install Cozystack on that cluster in one of two modes (install): As a Platform (pick a variant such as isp-full; Cozystack installs and configures all components) or BYOP (install the operator with the default variant that provides the package registry PackageSources, then use the cozypkg CLI). The operator is delivered as an OCI Helm chart: helm ... oci://ghcr.io/cozystack/cozystack/cozy-installer. (3) Configure via the cozystack ConfigMap (variant, pod/service CIDRs, root host, API endpoint, storage) and create a MetalLB IP pool.
• How an adopter tests/validates. Verify nodes Ready (kubectl get node), all platform HelmReleases reconciled (kubectl get hr -A — none failing), core pods Running, the aggregated API responds and the dashboard is reachable; then smoke-test by provisioning a sample tenant/app (e.g. a small Kubernetes cluster or Postgres) to Ready. Tenant clusters can be validated with upstream conformance (Sonobuoy) since Cozystack is Certified Kubernetes.

Security

• Security self-assessment. Cozystack has completed a Security Self-Assessment submitted with the Incubation application (cncf/toc#1916), reviewed with TAG Security. The assessment is maintained in-tree at docs/security/self-assessment.md, alongside a threat model and an incident-response process. Where this questionnaire and the self-assessment overlap, the self-assessment is the more detailed source and the two are kept consistent.
• Cloud Native Security Tenets.
  • How satisfied: secure-by-default (RBAC, per-tenant NetworkPolicies, isolated tenant API servers, immutable Talos nodes with no SSH/shell, minimal host attack surface); defense-in-depth via network segmentation (Cilium/Kube-OVN); automated TLS via cert-manager; secret management via External Secrets Operator/OpenBao; least-privilege service accounts for controllers; supply-chain hygiene via CodeQL, OpenSSF Scorecard and organization-wide Trivy CVE scanning of dependencies and container images.
  • Loosening defaults: adopters can relax tenant isolation (shared/flat networking), disable specific NetworkPolicies, or grant broader RBAC for single-tenant/lab use. [maintainers: link docs on loosening tenant-isolation/network-policy defaults]
• Security Hygiene.
  • Practices: GitHub review with required approvals; CI on every PR. The branch protection on main currently requires two status checks, pre-commit and E2E Tests. CodeQL static analysis runs on every PR, on push to main and weekly; new error-severity alerts are expected to be resolved or explicitly dismissed before merge, but CodeQL is not yet a required check — promoting it to one is an open item, and SECURITY.md states it as an intention rather than a fact. zizmor audits the GitHub Actions workflows themselves on PRs and fails the check on findings such as unpinned actions or over-broad permissions. OpenSSF Scorecard runs weekly and publishes to scorecard.dev; the OpenSSF Best Practices badge is maintained. Organization-wide Trivy scanning covers dependencies and container images on a 6-hour cycle for CRITICAL findings and weekly for the rest, with per-CVE triage tracked as issues (see Day 2 → Dependencies). Renovate automates dependency updates within the scope described in Day 2 → Dependencies. Component versions are pinned per release.
  • Unmaintained-feature risk: components are curated and version-pinned per release; the Helm-packaged, swappable building-block design makes replacing a risky/unmaintainable component tractable. [maintainers: describe the formal process for retiring/replacing a risky component]
• Cloud Native Threat Modeling.
  • Least privilege: controllers run with scoped RBAC; elevated privileges are limited to and documented for infrastructure components that genuinely need them (KubeVirt/CDI for virtualization, Piraeus/LINSTOR for block devices, Cilium/Kube-OVN for CNI). The published threat model is docs/security/threat-model.md.
  • Certificate rotation: automated via cert-manager (issuance/renewal/rotation) for platform TLS; Talos/Kubernetes-managed PKI for cluster components; tenant control-plane certs managed by Kamaji.
  • Secure software supply chain: in place today — CodeQL SAST (PRs, push to main, weekly), zizmor workflow auditing (gating on PRs), OpenSSF Scorecard (weekly), continuous dependency and container-image CVE scanning with Trivy across the whole organization (see Day 2 → Dependencies → SCA), Renovate-managed dependency updates, component versions pinned per release, and private vulnerability reporting with defined CVSS response targets.
  • Supply-chain gaps, stated precisely: artifact signing does not exist — there is no cosign or sigstore anywhere in the build tooling, and no issue tracks adding it. Build provenance is explicitly disabled: hack/common-envs.mk sets --provenance=false in BUILDX_ARGS, so the attestation buildx would emit by default is switched off, and no SLSA provenance is published. SBOM generation is implemented but off by default: the same file plumbs SBOM=1 through to --sbom=true, attaching a CycloneDX SBOM as a buildx attestation, and defaults it to 0 pending verification of registry referrers/attestation support; the remaining work is tracked in cozystack#2937 under the CI/build/release tracking issue cozystack#3278. Turning SBOMs on by default and re-enabling provenance are therefore configuration-and-verification work rather than greenfield work; signing is not.

---

Day 1 - Installation and Deployment Phase

Project Installation and Configuration

• As in Day 0 → Installation: Talos-based management cluster, then Cozystack via Platform or BYOP mode (OCI Helm installer + cozystack ConfigMap + MetalLB pool). All subsequent configuration is declarative (Kubernetes resources / Flux-managed HelmReleases); Day-2 changes are made by editing resources, not imperative steps.

Project Enablement and Rollback

• Enable/disable in a live cluster. Cozystack is the distribution rather than an add-on toggled inside a foreign cluster, so "enablement" = installing the operator bundle on the management cluster. Individual bundles/components and each tenant application are enabled/removed independently (create/delete the corresponding resource); removing a tenant application affects only that tenant, with no downtime to unrelated workloads. [maintainers: confirm whether any core component change requires control-plane restart]
• How enablement changes default cluster behavior. Installing Cozystack sets the CNI (Cilium/Kube-OVN), default StorageClass (LINSTOR), ingress, cert-manager, and the aggregated API as cluster defaults for subsequently created workloads. Cozystack expects to own the cluster, so a clean install does not disturb pre-existing unrelated workloads.
• Testing enablement/disablement. CI installs Cozystack end-to-end in ephemeral environments and provisions/destroys sample tenants and applications to validate create→ready→delete flows. Workflows: pull-requests.yaml (per-PR checks), e2e-tag.yaml (tag/release e2e), nightly.yaml (scheduled full run) and ui-test.yaml (dashboard). Test conventions are documented in docs/agents/e2e-testing.md. Current health, so that a reviewer following these links is not surprised: the per-PR checks are the gating ones and are healthy — pre-commit and E2E Tests are the two required status checks on main. The scheduled nightly.yaml run is presently red and has been failing since 2026-08-02, and the tag-triggered e2e-tag.yaml also failed around the v1.6.1 release; the scheduled-run instability is tracked in cozystack#3278. Enablement/disablement coverage therefore rests on the per-PR e2e job today, with the nightly full-matrix run being stabilised.
• Resource cleanup incl. CRDs. Deleting an application resource triggers Flux/Helm to remove its HelmRelease and owned objects; finalizers ensure dependent resources (PVCs, VM disks, LB IPs) are released. [maintainers: confirm CRD-cleanup behavior on full uninstall and any manual steps]

Rollout, Upgrade and Rollback Planning

• Compatibility with Kubernetes & frequency. Cozystack ships a curated, tested Kubernetes version per release; tenant clusters are Certified Kubernetes following upstream skew/support. Platform upgrades are delivered as new Cozystack releases. The authoritative list of what has shipped is the GitHub Releases page; which release lines are still receiving security fixes is defined as a rolling policy in SECURITY.md (latest stable minor and the one before it are supported; older v1.x lines get critical fixes at maintainer discretion; v0.x is out of support). Specific version numbers are deliberately not restated here, because they drift with every release.
• Upgrade procedure (upgrade docs): run the troubleshooting/health checklist; annotate the cozy-system namespace and cozystack-version ConfigMap with helm.sh/resource-policy=keep; helm upgrade cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer --version X.Y.Z (do not use --reuse-values — the chart pins OCI repo/package versions); verify pods and kubectl get hr -A.
• Rollback. GitOps/Helm-managed with versioned releases; roll back by targeting the previous release (previous HelmRelease revisions) via the same mechanism. Stateful data (databases, VM disks) persists on replicated storage across control-plane rollbacks. [maintainers: confirm a documented rollback runbook / version-skip constraints]
• How a rollout/rollback can fail & impact. Risks: incompatible CRD schema change, storage/CNI component upgrades requiring Talos node reboots, or a database-operator major bump. Impact is scoped to the component; running tenant workloads generally continue, though storage/CNI upgrades can transiently affect networking or volume attachment during node rolls. [maintainers: confirm whether upgrades trigger Talos node reboots]
• Metrics that should inform a rollback. Control-plane/API availability & latency, Flux reconciliation failures (kubectl get hr -A | grep -v True), DRBD/LINSTOR replication health (linstor resource list --faulty), CNI/pod-networking errors, tenant application readiness.
• Upgrade/rollback testing (upgrade→downgrade→upgrade). [maintainers: describe the CI matrix testing N-1→N upgrades and the downgrade path; link results]
• Deprecations/removals communication. Release notes/CHANGELOG, GitHub release announcements, docs and community channels; API deprecations follow Kubernetes-style windows. [maintainers: link the deprecation policy]
• Alpha/beta capabilities in a rollout. Experimental bundles/apps are opt-in and marked as such; only Stable/Patch releases are recommended for production. [maintainers: confirm how alpha/beta features are flagged per application]

---

Day 2 - Day-to-Day Operations Phase

Scalability/Reliability

• Increasing size/count of API objects. Scale horizontally by adding Talos nodes and by creating more tenants/applications (each an independent set of Flux-reconciled Kubernetes objects); storage scales by adding disks/nodes to the LINSTOR pool; Cluster Autoscaler and Vertical Pod Autoscaler assist node/pod right-sizing.
• SLOs/SLIs. [maintainers: state whether the project publishes reference SLOs/SLIs for the management API, tenant control-plane availability and storage, or note that SLOs are adopter-defined and list the SLIs exposed]
• Operations increasing in time under existing SLIs. Provisioning large tenant clusters, VM live-migration, and DRBD full-resync after node replacement scale with size/data volume. [maintainers: confirm]
• Resource-usage increase from enabling the project. Platform components (Cilium, Kube-OVN, KubeVirt/CDI, LINSTOR, VictoriaMetrics/VictoriaLogs, Flux, DBaaS operators) consume a baseline on the management cluster; VM/DB workloads dominate at scale. The hardware tiers above bound expected footprints. [maintainers: provide measured baseline overhead of platform components]
• Conditions causing node-resource exhaustion. Many VMs/pods per node (PIDs, sockets, hugepages), inodes on storage nodes, DRBD/LINSTOR memory under heavy replication, conntrack under high connection counts. Quotas/limits, VPA and MetalLB/ingress sizing mitigate. [maintainers: confirm]
• Load testing performed & results. [maintainers: summarize scale/load tests — nodes, tenants, VMs, DB throughput — and link results; known gap to fill for Incubation DD]
• Recommended limits & how obtained. Indicative per-tier limits are published in the hardware requirements (e.g. large-production ≈ 20+ tenants, dozens of clusters, hundreds of VMs/DBs on 6+ nodes). [maintainers: provide tested maximums (nodes, tenants, VMs, PVs) and methodology]
• Resilience patterns. Controller reconcile loops with retries/back-off (Kubernetes/Flux), synchronous storage replication (DRBD), leader election, health probes and automatic rescheduling, MetalLB failover, cert-manager auto-renewal. [maintainers: note any explicit circuit-breaker usage, e.g. at ingress]

Observability Requirements

• Signals produced (monitoring docs): metrics via VictoriaMetrics (VMCluster) collected by VMAgent (Prometheus-compatible); logs via VictoriaLogs (VLogs) collected by Fluent Bit (DaemonSet); dashboards via Grafana; alerting via Alerta; plus kube-state-metrics and node-exporter. Kubernetes events and component logs in standard formats.
• Audit logging. Kubernetes API audit logging can be enabled on management and tenant control planes through the standard kube-apiserver audit policy. No default audit policy is shipped by Cozystack — no audit-policy manifest exists in any package — so audit logging is opt-in and the policy is authored by the adopter. Enabling it by default, with a documented reference policy, is an open improvement rather than an existing feature.
• Dashboards. Pre-built Grafana dashboards for cluster overview, node metrics, storage, KubeVirt/VMs and per-application health, plus user-created dashboards (Grafana Operator manages dashboards as CRDs).
• Cost/FinOps surfacing. Per-tenant/per-application resource usage is observable via metrics and Kubernetes resource accounting (chargeback/showback), integrating with provider billing/portal. [maintainers: link any FinOps/billing integration]
• Health parameters. Control-plane/API availability, Flux reconciliation status, Talos/node health, DRBD replication state, pod/VM readiness, ingress/LB health.
• Determining the project is in use. Presence of Cozystack CRDs/aggregated APIs (apps.cozystack.io) and reconciled platform HelmReleases; existence of Tenant/application resources.
• Knowing it works for your instance. Dashboards green; all HelmReleases Ready; sample tenant/app provisions successfully; storage replication healthy.
• Per-tenant monitoring. Tenant workloads connect custom exporters via VMServiceScrape/VMPodScrape.

Dependencies

• Running services depended on. As listed in Day 0 → in-cluster service dependencies (full platform stack).
• Dependency lifecycle policy. Component versions are pinned and shipped per Cozystack release; upgrades are curated and tested together and bumped as a single coordinated release. Renovate (.github/renovate.json) automates part of this, and it is worth being exact about which part, because the automated share is smaller than the phrase "Renovate-managed" usually implies. Its enabledManagers are gomod, dockerfile, github-actions and custom.regex; the helm-values manager is disabled repo-wide, because many image references in the vendored charts are assembled at render time by the cozy-lib.image template helper rather than written literally in values.yaml, so a values-file manager cannot see them. Two bespoke regex managers cover specific pins that would otherwise age silently. Consequently: Go modules, Dockerfile base images, GitHub Actions and those regex-pinned refs get automated PRs, while the curated component images that make up the shipped platform are bumped by maintainers as part of preparing a release, not by a bot. That is a deliberate consequence of shipping a tested, coordinated version set, but it does mean the dependency class that dominates the product is on human cadence.
• Source Composition Analysis (SCA). Four layers. (1) Static analysis of first-party code — CodeQL on every PR, on push to main and weekly, reporting only alerts new relative to main; expected to be resolved or dismissed before merge, though not yet a required check. (2) Static analysis of the CI supply chain itself — zizmor audits the GitHub Actions workflows on PRs and fails the check on unpinned actions, over-broad permissions and similar findings. (3) Supply-chain posture — OpenSSF Scorecard weekly, published on scorecard.dev, plus the OpenSSF Best Practices badge. (4) Dependency and container-image CVE scanning — an organization-wide pipeline built on Trivy running against every non-fork, non-archived repository in the cozystack organization.
• How the CVE-scanning pipeline works. Discovery walks each repository for Go modules (go.mod, direct dependencies), base images from Dockerfile FROM directives, and container images referenced in values.yaml under packages/apps/*, packages/system/* and packages/core/*. Trivy then scans both the repository filesystem — which is where transitive Go dependencies and lock files are actually covered, since discovery itself reads only direct requirements — and every resolved container image, against NVD, vendor advisories and the GitHub Advisory Database. A known limitation: image references assembled at render time by the cozy-lib.image helper are not literal strings in values.yaml and are therefore invisible to values-file discovery, the same blind spot that led to disabling Renovate's helm-values manager. Those images are covered only insofar as they appear elsewhere; closing the gap requires resolving refs through the rendered chart rather than the values file. CRITICAL findings are scanned and reported every 6 hours; HIGH/MEDIUM/LOW are collected into a weekly report. Four filters run before an issue is created — findings with no upstream fix older than 365 days, dev and build-only dependencies (golang:/node:/python: images, core/testing, hack/, e2e/), CVEs already carrying a maintainer triage decision, and findings with no severity assigned in any database — so every new actionable CVE becomes a tracked issue with severity labels, affected components, fix version and a triage checklist, while roughly 20–30% of raw output is suppressed as noise. Triage decisions are persisted, so a settled finding is not re-reported.
• Public verifiability of the pipeline — an open problem, stated plainly. The pipeline, its reports and its triage state live in cozystack/security-scanner, which returns 404 to anyone outside the organization. Only two parts of it genuinely require confidentiality: the per-CVE reports and the triage-state files, which describe unfixed exposure in shipped artifacts and would function as a target list for adopters running affected releases. The rest — the pipeline policy (SECURITY_PIPELINE_POLICY.md), the severity/SLA matrix (SECURITY_TRIAGE_MATRIX.md), the finding status model (SECURITY_FINDING_STATUS_MODEL.md), the scanning scripts, and the monthly aggregate summaries — contains no vulnerability detail and is what actually evidences the control. The project's intent is therefore to publish those and keep only per-CVE findings and triage state embargoed until fixed, so that this answer is verifiable without privileged access; the embargo scope is per-finding, not per-repository. [maintainers: this publication needs to happen before submission, not be promised in it — it is a maintainer action, and until it lands a TOC reviewer cannot confirm any of the above]
• Acting on SCA & timescale. Two tracks with distinct targets. Reported vulnerabilities (Cozystack-specific code, packaging, defaults, integration logic) follow SECURITY.md: acknowledgement ≤3 business days, initial triage ≤7 business days, remediation targets Critical ≈14 days, High ≈30 days, Medium ≈90 days, Low next release, under a coordinated-disclosure window of up to 90 days. Scanner findings in bundled upstream components are a separate track with a tighter triage clock: Critical triaged within 1 business day against a 7-day fix target, High within 3 business days / 30 days, Medium within 10 business days / 90 days, Low within 30 days best-effort. Triage follows an explicit decision tree (is the CVE in a shipped artifact rather than dev/test tooling; is the vulnerable path reachable in Cozystack's usage; has the distribution already backported the fix; is an upstream fix available) and resolves to one of confirmed, false-positive, accepted-risk, in-progress or fixed. Confirmed findings ship as a pinned-component version bump in the next Cozystack release, or as a patch release on a supported line when severity warrants it. These are documented targets, not measured attainment. The scanner track is currently running a triage backlog against its stated clock, and the rotating role that owns meeting it is defined but unfilled. [maintainers: supply current time-to-triage figures and open-backlog counts to the TOC through the private channel, together with the remediation plan — TAG Security asks for the distribution, and publishing raw open-CRITICAL counts in this repository would itself disclose unfixed exposure]

Troubleshooting

• Recovery when a key component fails (troubleshooting): Kubernetes/etcd HA and Talos self-healing recover control-plane node loss; Kamaji reschedules tenant control planes; DRBD/LINSTOR promotes a healthy replica; Flux re-reconciles drifted/removed platform objects; databases recover via their operators; Velero provides backup/restore. Diagnostics: kubectl get node; kubectl get hr -A | grep -v True; kubectl get pod -A | grep -v 'Running\|Completed'; linstor node/storage-pool/resource list [--faulty]; operator logs kubectl logs -n cozy-system deploy/cozystack-operator; dedicated guides for etcd, Flux CD and Kube-OVN.
• Known failure modes. Documented areas include broken HelmReleases (DependenciesNotReady when a required package is disabled/missing), LINSTOR/DRBD faulty resources, Kube-OVN OVN-database corruption (recover via ovn-appctl procedures), and etcd quorum issues — each with a dedicated troubleshooting guide. [maintainers: consolidate a "known failure modes + mitigations" list if not already published]

Compliance

• Third-party attribution & license notices. The project is Apache-2.0 with a top-level LICENSE, and bundles third-party components under their own licenses, whose license files travel with the vendored component. There is no top-level NOTICE file. For a distribution that ships on the order of fifty upstream components, that is a real compliance gap rather than an oversight to paper over: adding a NOTICE that aggregates attribution for the bundled components, and a documented process for keeping it in step with the component set, is outstanding work.
• Alignment with CNCF attribution recommendations.
  • Notices for third-party code in source files: retained in-file / in LICENSE/NOTICE. [maintainers: confirm]
  • Notices for unmodified third-party components in the repo: retained alongside the pinned component. [maintainers: confirm]
  • Notices in distributed build artifacts (images/binaries): [maintainers: describe how license notices/SBOM are included in container images]

Security

• Access control execution. Kubernetes RBAC end-to-end; per-tenant namespaces, RoleBindings, quotas and NetworkPolicies; isolated tenant API servers (Kamaji + dedicated etcd); Keycloak OIDC/SAML for federated auth; secrets via External Secrets Operator / OpenBao.
• Security reporting team & rotation. Reporting via GitHub Private Vulnerability Reporting (advisories/new) or a private maintainer contact; any maintainer can receive and route a report. Current security response team: @kvaps, @lexfrei, @tym83, @matthieu-robin, @mattia-eleuteri. [maintainers: describe how security-team membership reflects community/organizational diversity and how members are invited/rotated — this rotation policy is not yet documented in SECURITY.md]

---

Appendix — sources

• cozystack.io docs v1.6: platform-stack, hardware-requirements, install, upgrade, monitoring, troubleshooting, OIDC, REST API.
• Repo (main): GOVERNANCE.md, CONTRIBUTOR_LADDER.md, MAINTAINERS.md, .github/CODEOWNERS, SECURITY.md, ADOPTERS.md.
• Releases: GitHub Releases.
• CI / security workflows: pull-requests.yaml, e2e-tag.yaml, nightly.yaml, ui-test.yaml, codeql.yml, scorecard.yml, renovate.json.
• Roadmap: GitHub Project board; supply-chain items cozystack#2937, cozystack#3278.
• SCA tooling: Trivy (organization-wide CVE scanning pipeline; the repository hosting it is private, see Day 2 → Dependencies).
• SSA / Incubation: cncf/toc#1916.
• GTR template: cncf/toc toc_subprojects/project-reviews-subproject/general-technical-questions.md v1.0.