cozystack/docs/agents/overview.md

Cozystack Project Overview

This document provides detailed information about Cozystack project structure and conventions for AI agents.

About Cozystack

Cozystack is an open-source Kubernetes-based platform and framework for building cloud infrastructure. It provides:

• Managed Services: Databases, VMs, Kubernetes clusters, object storage, and more
• Multi-tenancy: Full isolation and self-service for tenants
• GitOps-driven: FluxCD-based continuous delivery
• Modular Architecture: Extensible with custom packages and services
• Developer Experience: Simplified local development with cozyhr tool

The platform exposes infrastructure services via the Kubernetes API with ready-made configs, built-in monitoring, and alerts.

Code Layout

.
├── packages/               # Main directory for cozystack packages
│   ├── core/              # Core platform logic charts (installer, platform)
│   ├── system/            # System charts (CSI, CNI, operators, etc.)
│   ├── apps/              # User-facing charts shown in dashboard catalog
│   └── extra/             # Tenant-specific modules, singleton charts which are used as dependencies
├── dashboards/            # Grafana dashboards for monitoring
├── hack/                  # Helper scripts for local development
│   └── e2e-chainsaw/     # End-to-end application tests (Kyverno Chainsaw)
├── scripts/               # Scripts used by cozystack container
│   └── migrations/       # Version migration scripts
├── docs/                  # Documentation
│   ├── agents/           # AI agent instructions
│   └── changelogs/       # Release changelogs
├── cmd/                   # Go command entry points
│   ├── cozystack-api/
│   ├── cozystack-controller/
│   └── cozystack-assets-server/
├── internal/              # Internal Go packages
│   ├── controller/       # Controller implementations
│   └── lineagecontrollerwebhook/
├── pkg/                   # Public Go packages
│   ├── apis/
│   ├── apiserver/
│   └── registry/
└── api/                   # Kubernetes API definitions (CRDs)
    └── v1alpha1/

Package Structure

Every package is a Helm chart following the umbrella chart pattern:

packages/<category>/<package-name>/
├── Chart.yaml                           # Chart definition and parameter docs
├── Makefile                             # Development workflow targets
├── charts/                              # Vendored upstream charts
├── images/                              # Dockerfiles and image build context
├── patches/                             # Optional upstream chart patches
├── templates/                           # Additional manifests
├── templates/dashboard-resourcemap.yaml # Dashboard resource mapping
├── values.yaml                          # Override values for upstream
└── values.schema.json                   # JSON schema for validation and UI

Build and Development Commands

Root targets (run from the repo root):

make build          # Build all Docker images (needs: docker, skopeo, jq, gh, helm, yq, GNU tar/sed/awk)
make unit-tests     # Run unit tests (requires bats-core >= 1.5; GNU parallel is optional)
make test-controllers # Run the Go tests under ./internal/... (controllers, contract tests)
make generate       # Code generation (hack/update-codegen.sh) — CRDs, DeepCopy, clients, RBAC
make manifests      # Generate CRD manifests and operator YAML variants
make cozypkg        # Build the cozypkg CLI
make test           # Full E2E tests (requires a cluster)
make prepare-env    # Prepare the E2E test environment (apply + prepare-cluster)

Per-package targets (run inside packages/{apps,system,extra,core}/<name>/):

make image          # Build the package image
make show           # Render the Helm template
make apply          # Apply the Helm release to a cluster (needs NAME, NAMESPACE)
make diff           # Diff the template against the cluster
make test           # Package-specific tests
make generate       # Regenerate values.schema.json + README.md from values.yaml

Build environment variables:

• REGISTRY — Docker registry (default ghcr.io/cozystack/cozystack).
• PUSH=1 / LOAD=0 — control buildx push/load behaviour.
• LOAD=1 PUSH=0 — load images locally instead of pushing. Builds for the host architecture only, because the classic docker image store cannot load a multi-platform index.
• PLATFORM — the buildx platforms. Defaults to linux/amd64,linux/arm64 for anything but LOAD=1, so a push from either architecture publishes an index both can run. Needs a docker-container builder (BUILDER) or a docker daemon on the containerd image store, with QEMU registered for the non-native half. Compiling builder stages run natively on $BUILDPLATFORM and cross-compile for the target, but every RUN step of a stage that runs on the target platform (package installs, setcap, install scripts, patch steps) still runs emulated. packages/core/testing pins linux/amd64, because it carries an x86 KVM sandbox. packages/core/talos builds the Talos installer and release assets for the arches in PLATFORM with the upstream imager, which needs no QEMU for either; its matchbox image carries the amd64 and arm64 kernel and initramfs whatever PLATFORM says, because it network-boots machines of either arch.
• PUSHED_TAGS_LOG — a file the image-tags macro appends every pushed <repo>:<tag> to, one per line. Unset by default; the release build sets it so hack/stitch-multiarch.sh can move every tag it pushed, versioned ones included, onto the multi-arch index. The append happens at recipe expansion, so make -n writes it too.
• CACHE_TAG — the tag of the mode=max build cache each image reads (and writes under WRITE_CACHE=1). Defaults to buildcache. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under <tag>-arm64 with CACHE_TAG=buildcache-arm64, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release, hack/stitch-multiarch.sh then joins each pair into one index, hack/verify-multiarch.sh fails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; see docs/release/ci-pipeline.md. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (see docs/agents/image-refs.md).

Values schema generation

Package values.yaml files carry annotations (@param, @typedef, @field, @enum, @section) that cozyvalues-gen turns into values.schema.json and README.md. Run make generate in the package after editing annotations. A pre-commit hook runs make generate across packages/apps/* and packages/extra/* on every commit to keep these in sync — see contributing.md for the regenerate-before-commit rule.

A chart also declares the limits on its own resource name there, with ## @name {string} - <why> followed by the constraint tags (@maxLength, @minLength, @pattern) it needs. cozyvalues-gen publishes the declaration at the root of values.schema.json as x-cozystack-name, from where hack/update-crd.sh carries it into the ApplicationDefinition, and the aggregated API server enforces it on metadata.name for whatever kind the definition describes. The description is the message a rejected name is answered with, so write it as the explanation of the limit. A @pattern is unanchored, as JSON Schema specifies, so one meant for the whole name is written ^...$. A declaration the server cannot enforce as written — a constraint keyword it does not implement, a wrong type, a null, lengths no name satisfies, a pattern that is not RE2 — refuses every create of that kind with a configuration error rather than being dropped. A pattern is checked only to compile, not to match anything, so one that matches no name ($^) rejects every name with the pattern as the reason. Annotation keywords such as title or examples, and x- extensions, are ignored. A cap only ever tightens: prefix + name must still fit the 53-character Helm release name limit, and a declaration looser than that budget does not widen it. The limits belong in the chart because that is where the facts behind them are — which sub-resources it renders and what release names they take — and a chart that needs nothing beyond the platform-wide limits declares nothing.

Testing

• Helm and BATS unit tests: make helm-unit-tests runs hack/helm-unit-tests.sh over every package that defines a test target. A test target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as packages/core/testing already does with its sandbox flow. make bats-unit-tests requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with pre-commit run bats-unit-tests --hook-stage manual --all-files; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose /bin/sh differs from CI, make BATS_POSIX_SHELL=dash bats-posix-compat-tests selects the same interpreter explicitly. make unit-tests runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under ./internal/...; run make test-controllers as well. CI invokes both targets together.
• E2E tests: Kyverno Chainsaw suites in hack/e2e-chainsaw/ (one directory per app), run with chainsaw test. Cluster bootstrap (hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in e2e-testing.md.
• Go tests: standard go test, with Ginkgo/Gomega for controllers.

Conventions

Helm Charts

• Follow umbrella chart pattern for system components
• Include upstream charts in charts/ directory (vendored, not referenced)
• Override configuration in root values.yaml
• Use values.schema.json for input validation and dashboard UI rendering

No lookup that copies another object into the render. A lookup reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one.

A lookup is fine when a repeated render gives the same output and carries nothing from another object into it: checking that an object exists, keeping a generated Secret's value, keeping an immutable field of the chart's own live object, or a guard that fails on an unsafe change. Keep its empty branch rendering, because offline helm template and helm-unittest without a kubernetesProvider mock see only that branch; test the other branch with such a mock.

To use a fact about the cluster instead, read it from values: _cluster and _namespace in the cozystack-values Secret, or valuesFrom on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with .Capabilities.APIVersions.Has, since a lookup on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object.

Go Code

• Follow standard Go conventions and idioms
• Use controller-runtime patterns for Kubernetes controllers
• Prefer kubebuilder for API definitions and controllers
• Add proper error handling and structured logging

Git Commits

Conventional Commits (type(scope): description) with --signoff, kept atomic and focused. Full format, scopes, AI-attribution trailer, and PR workflow are in contributing.md — the source of truth; do not duplicate them here.

PackageSource CRD upgrade policy

Each component in a PackageSource may set install.upgradeCRDs to control how CRDs from the chart's crds/ directory are handled on HelmRelease upgrades. Allowed values: Skip (default — helm-controller does not touch CRDs on upgrade), Create (create new CRDs only), CreateReplace (create new and overwrite existing).

Set upgradeCRDs: CreateReplace for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them.

Do not set CreateReplace blindly: it overwrites every CRD in crds/ and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually.

Documentation — where a new document goes

Product documentation lives in cozystack/website, not here. This repository holds only documentation that something in the repository references by path, or that describes the repository's own machinery. Pick the destination before writing, because moving a document later means chasing every @param, template comment and test comment that points at it.

Destination	What belongs there
packages/<group>/<name>/README.md	Anything scoped to a single package. Prose goes above ## Parameters; that section is cozyvalues-gen output and is never hand-edited. The website's make update-apps / update-vms / update-networking / update-k8s / update-services targets append the whole README onto a published page, but only for the packages named in the APPS, VMS, NETWORKING, K8S and SERVICES lists of the website Makefile, and only from packages/apps/ and packages/extra/: nothing under packages/system/ is ever imported, and neither is a package that is not on those lists. The sync runs when a release is promoted or tagged, or by hand in the website repository, not when a pull request merges here. Use this route only for a package that is on the lists; for any other the prose reaches no reader, so put it on the website instead.
cozystack/website, under content/en/docs/	Operator- and tenant-facing content that spans packages or is task-shaped: guides, how-tos, troubleshooting runbooks, reference tables. Which version directory a page goes into is set by the website's own CONTRIBUTING.md; follow that file rather than a copy of its rules here.
docs/	Documentation that a chart template, migration, script or workflow references by repo-relative path, plus this repository's own process docs (docs/agents/, docs/changelogs/, docs/release.md and the pages under docs/release/).
api/<group>/<version>/DESIGN.md, pkg/apis/<group>/DESIGN.md	Design of an API group, beside the types it describes.
cozystack/community	Design proposals and RFCs. Not this repository.

When a document could go in either docs/ or the website, the test is whether a non-documentation file has to link to it by path. docs/operations/seaweedfs-431-rename-recovery.md stays because packages/*/seaweedfs/templates/_naming.tpl and a platform migration point at it. A guide referenced only from a values.yaml @param does not qualify — put it on the website and link the URL.

Documenting a feature's current limitations is useful and belongs with the feature. Documenting the development process around it does not: no "What is NOT in Phase N", no "out of scope for this PR", no "this PR" phrasing, no action plans or review negotiation. A merged document describes what the code does now, and states a limitation as a limitation rather than as a roadmap position. Scope, phasing and follow-ups belong in the pull request description or an issue.

A document that names an unmerged pull request or an unreleased dependency needs a tracking issue, because nothing re-checks that claim later and it silently becomes false once the thing ships.

Domain references for specific tasks

Before working on these areas, read the relevant doc first:

• Backup subsystem design: api/backups/v1alpha1/DESIGN.md
• A new app package: copy the structure of an existing one (e.g. packages/apps/postgres/)

Things Agents Should Not Do

Never Edit These

• Do not modify files in /vendor/ (Go dependencies)
• Do not edit generated files: zz_generated.*.go
• Do not change go.mod/go.sum manually (use go get)
• Do not edit upstream charts in packages/*/charts/ directly (use patches)
• Do not modify image digests in values.yaml (generated by build)

Version Control

• Do not commit built artifacts from _out
• Do not commit test artifacts or temporary files

Git Operations

• Do not force push to main/master
• Do not update git config
• Do not perform destructive operations without explicit request

Core Components

• Do not modify packages/core/platform/ without understanding migration impact