cozystack/docs/agents/overview.md

Cozystack Project Overview

This document provides detailed information about Cozystack project structure and conventions for AI agents.

About Cozystack

Cozystack is an open-source Kubernetes-based platform and framework for building cloud infrastructure. It provides:

• Managed Services: Databases, VMs, Kubernetes clusters, object storage, and more
• Multi-tenancy: Full isolation and self-service for tenants
• GitOps-driven: FluxCD-based continuous delivery
• Modular Architecture: Extensible with custom packages and services
• Developer Experience: Simplified local development with cozyhr tool

The platform exposes infrastructure services via the Kubernetes API with ready-made configs, built-in monitoring, and alerts.

Code Layout

.
├── packages/               # Main directory for cozystack packages
│   ├── core/              # Core platform logic charts (installer, platform)
│   ├── system/            # System charts (CSI, CNI, operators, etc.)
│   ├── apps/              # User-facing charts shown in dashboard catalog
│   └── extra/             # Tenant-specific modules, singleton charts which are used as dependencies
├── dashboards/            # Grafana dashboards for monitoring
├── hack/                  # Helper scripts for local development
│   └── e2e-chainsaw/     # End-to-end application tests (Kyverno Chainsaw)
├── scripts/               # Scripts used by cozystack container
│   └── migrations/       # Version migration scripts
├── docs/                  # Documentation
│   ├── agents/           # AI agent instructions
│   └── changelogs/       # Release changelogs
├── cmd/                   # Go command entry points
│   ├── cozystack-api/
│   ├── cozystack-controller/
│   └── cozystack-assets-server/
├── internal/              # Internal Go packages
│   ├── controller/       # Controller implementations
│   └── lineagecontrollerwebhook/
├── pkg/                   # Public Go packages
│   ├── apis/
│   ├── apiserver/
│   └── registry/
└── api/                   # Kubernetes API definitions (CRDs)
    └── v1alpha1/

Package Structure

Every package is a Helm chart following the umbrella chart pattern:

packages/<category>/<package-name>/
├── Chart.yaml                           # Chart definition and parameter docs
├── Makefile                             # Development workflow targets
├── charts/                              # Vendored upstream charts
├── images/                              # Dockerfiles and image build context
├── patches/                             # Optional upstream chart patches
├── templates/                           # Additional manifests
├── templates/dashboard-resourcemap.yaml # Dashboard resource mapping
├── values.yaml                          # Override values for upstream
└── values.schema.json                   # JSON schema for validation and UI

Build and Development Commands

Root targets (run from the repo root):

make build          # Build all Docker images (needs: docker, skopeo, jq, gh, helm, yq, GNU tar/sed/awk)
make unit-tests     # Run unit tests (requires bats-core >= 1.5; GNU parallel is optional)
make test-controllers # Run the Go tests under ./internal/... (controllers, contract tests)
make generate       # Code generation (hack/update-codegen.sh) — CRDs, DeepCopy, clients, RBAC
make manifests      # Generate CRD manifests and operator YAML variants
make cozypkg        # Build the cozypkg CLI
make test           # Full E2E tests (requires a cluster)
make prepare-env    # Prepare the E2E test environment (apply + prepare-cluster)

Per-package targets (run inside packages/{apps,system,extra,core}/<name>/):

make image          # Build the package image
make show           # Render the Helm template
make apply          # Apply the Helm release to a cluster (needs NAME, NAMESPACE)
make diff           # Diff the template against the cluster
make test           # Package-specific tests
make generate       # Regenerate values.schema.json + README.md from values.yaml

Build environment variables:

• REGISTRY — Docker registry (default ghcr.io/cozystack/cozystack).
• PUSH=1 / LOAD=0 — control buildx push/load behaviour.
• LOAD=1 PUSH=0 — load images locally instead of pushing. Builds for the host architecture only, because the classic docker image store cannot load a multi-platform index.
• PLATFORM — the buildx platforms. Defaults to linux/amd64,linux/arm64 for anything but LOAD=1, so a push from either architecture publishes an index both can run. Needs a docker-container builder (BUILDER) or a docker daemon on the containerd image store, with QEMU registered for the non-native half. Compiling builder stages run natively on $BUILDPLATFORM and cross-compile for the target, but every RUN step of a stage that runs on the target platform (package installs, setcap, install scripts, patch steps) still runs emulated. packages/core/testing pins linux/amd64, because it carries an x86 KVM sandbox. packages/core/talos builds the Talos installer and release assets for the arches in PLATFORM with the upstream imager, which needs no QEMU for either; its matchbox image carries the amd64 and arm64 kernel and initramfs whatever PLATFORM says, because it network-boots machines of either arch.
• PUSHED_TAGS_LOG — a file the image-tags macro appends every pushed <repo>:<tag> to, one per line. Unset by default; the release build sets it so hack/stitch-multiarch.sh can move every tag it pushed, versioned ones included, onto the multi-arch index. The append happens at recipe expansion, so make -n writes it too.
• CACHE_TAG — the tag of the mode=max build cache each image reads (and writes under WRITE_CACHE=1). Defaults to buildcache. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under <tag>-arm64 with CACHE_TAG=buildcache-arm64, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release, hack/stitch-multiarch.sh then joins each pair into one index, hack/verify-multiarch.sh fails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; see docs/release.md. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (see docs/agents/image-refs.md).

Values schema generation

Package values.yaml files carry annotations (@param, @typedef, @field, @enum, @section) that cozyvalues-gen turns into values.schema.json and README.md. Run make generate in the package after editing annotations. A pre-commit hook runs make generate across packages/apps/* and packages/extra/* on every commit to keep these in sync — see contributing.md for the regenerate-before-commit rule.

A chart also declares the limits on its own resource name there, with ## @name {string} - <why> followed by the constraint tags (@maxLength, @minLength, @pattern) it needs. cozyvalues-gen publishes the declaration at the root of values.schema.json as x-cozystack-name, from where hack/update-crd.sh carries it into the ApplicationDefinition, and the aggregated API server enforces it on metadata.name for whatever kind the definition describes. The description is the message a rejected name is answered with, so write it as the explanation of the limit. A @pattern is unanchored, as JSON Schema specifies, so one meant for the whole name is written ^...$. A declaration the server cannot enforce as written — a constraint keyword it does not implement, a wrong type, a null, lengths no name satisfies, a pattern that is not RE2 — refuses every create of that kind with a configuration error rather than being dropped. A pattern is checked only to compile, not to match anything, so one that matches no name ($^) rejects every name with the pattern as the reason. Annotation keywords such as title or examples, and x- extensions, are ignored. A cap only ever tightens: prefix + name must still fit the 53-character Helm release name limit, and a declaration looser than that budget does not widen it. The limits belong in the chart because that is where the facts behind them are — which sub-resources it renders and what release names they take — and a chart that needs nothing beyond the platform-wide limits declares nothing.

Testing

• Helm and BATS unit tests: make helm-unit-tests runs hack/helm-unit-tests.sh over every package that defines a test target. A test target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as packages/core/testing already does with its sandbox flow. make bats-unit-tests requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with pre-commit run bats-unit-tests --hook-stage manual --all-files; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose /bin/sh differs from CI, make BATS_POSIX_SHELL=dash bats-posix-compat-tests selects the same interpreter explicitly. make unit-tests runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under ./internal/...; run make test-controllers as well. CI invokes both targets together.
• E2E tests: Kyverno Chainsaw suites in hack/e2e-chainsaw/ (one directory per app), run with chainsaw test. Cluster bootstrap (hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in e2e-testing.md.
• Go tests: standard go test, with Ginkgo/Gomega for controllers.

Test first, failure path first

1. Write the test before the change and watch it fail. A test that has never been red proves nothing, and the red run must fail for the reason the test names, not on a typo or a missing fixture. Then make it green, then refactor under it.
2. Cover the failure path before the happy path. Every user exercises the happy path; the failure path usually runs for the first time in production. Start with an empty or null value, invalid input, a number where a string is expected, a missing resource, a timeout, an API error, and pin what the code does then: the message it prints, the status it exits with, the state it leaves untouched.
3. A red test is fixed in the code, not in the test. Do not skip, weaken or delete a test, or silence a linter, to get a green run.
4. A stub (a not-implemented branch, an empty implementation) lands together with a test that fails on it and states the behaviour expected once it is done.
5. A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked FIXME(#N). The PR that fixes the bug flips the test to pin the fixed behaviour.

A negated BATS assertion can never be red; see §11 of e2e-testing.md.

Linters

Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline.

Markdown follows these markdownlint rules: every code fence names a language (text when nothing else fits), table separators have spaces around the pipes (| --- |), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as Prose Formatting explains.

Conventions

Helm Charts

• Follow umbrella chart pattern for system components
• Include upstream charts in charts/ directory (vendored, not referenced)
• Override configuration in root values.yaml
• Use values.schema.json for input validation and dashboard UI rendering
• An unquoted numeric value (name: 123) reaches a template as a number: len on it aborts the render with len of type float64 (int64 for a value given with --set) before any fail message is reached, and if not $v treats 0 as unset. Coerce with printf "%v" before validating a value that may be an unquoted scalar, and test for emptiness with eq $v "" after the coercion. A YAML null or a missing key coerces to the string <nil>, which that test reports as non-empty, so reject it first with kindIs "invalid" on the raw value, before printf

Container images

A new first-party image is built from a Containerfile, which the package Makefile passes to docker buildx build with --file. Existing Dockerfiles and the files of vendored images stay as they are.

Examples and fixtures

Addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24), IPv6 from RFC 3849 (2001:db8::/32), domains from RFC 2606 and RFC 6761 (example.com, .example, .test, .invalid), ASNs from RFC 5398 (64496–64511, 65536–65551), MAC addresses from RFC 7042 (00:00:5E:00:53:00–00:00:5E:00:53:FF). Where no reserved range exists, use an obviously synthetic placeholder such as <NODE_IP>.

Go Code

• Follow standard Go conventions and idioms
• Use controller-runtime patterns for Kubernetes controllers
• Prefer kubebuilder for API definitions and controllers
• Add proper error handling and structured logging

Git Commits

Conventional Commits (type(scope): description) with --signoff, kept atomic and focused. Full format, scopes, AI-attribution trailer, and PR workflow are in contributing.md — the source of truth; do not duplicate them here.

PackageSource CRD upgrade policy

Each component in a PackageSource may set install.upgradeCRDs to control how CRDs from the chart's crds/ directory are handled on HelmRelease upgrades. Allowed values: Skip (default — helm-controller does not touch CRDs on upgrade), Create (create new CRDs only), CreateReplace (create new and overwrite existing).

Set upgradeCRDs: CreateReplace for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them.

Do not set CreateReplace blindly: it overwrites every CRD in crds/ and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually.

Documentation

Documentation is organized as follows:

• docs/ - General documentation
• docs/agents/ - Instructions for AI agents
• docs/changelogs/ - Release changelogs
• Main website: https://github.com/cozystack/website

Domain references for specific tasks

Before working on these areas, read the relevant doc first:

• Backup subsystem design: api/backups/v1alpha1/DESIGN.md
• A new app package: copy the structure of an existing one (e.g. packages/apps/postgres/)

Things Agents Should Not Do

Never Edit These

• Do not modify files in /vendor/ (Go dependencies)
• Do not edit generated files: zz_generated.*.go
• Do not change go.mod/go.sum manually (use go get)
• Do not edit upstream charts in packages/*/charts/ directly (use patches)
• Do not modify image digests in values.yaml (generated by build)

Version Control

• Do not commit built artifacts from _out
• Do not commit test artifacts or temporary files

Git Operations

• Do not force push to main/master
• Do not update git config
• Do not perform destructive operations without explicit request

Core Components

• Do not modify packages/core/platform/ without understanding migration impact