# Cozystack Project Overview This document provides detailed information about Cozystack project structure and conventions for AI agents. ## About Cozystack Cozystack is an open-source Kubernetes-based platform and framework for building cloud infrastructure. It provides: - **Managed Services**: Databases, VMs, Kubernetes clusters, object storage, and more - **Multi-tenancy**: Full isolation and self-service for tenants - **GitOps-driven**: FluxCD-based continuous delivery - **Modular Architecture**: Extensible with custom packages and services - **Developer Experience**: Simplified local development with cozyhr tool The platform exposes infrastructure services via the Kubernetes API with ready-made configs, built-in monitoring, and alerts. ## Code Layout ``` . ├── packages/ # Main directory for cozystack packages │ ├── core/ # Core platform logic charts (installer, platform) │ ├── system/ # System charts (CSI, CNI, operators, etc.) │ ├── apps/ # User-facing charts shown in dashboard catalog │ └── extra/ # Tenant-specific modules, singleton charts which are used as dependencies ├── dashboards/ # Grafana dashboards for monitoring ├── hack/ # Helper scripts for local development │ └── e2e-chainsaw/ # End-to-end application tests (Kyverno Chainsaw) ├── scripts/ # Scripts used by cozystack container │ └── migrations/ # Version migration scripts ├── docs/ # Documentation │ ├── agents/ # AI agent instructions │ └── changelogs/ # Release changelogs ├── cmd/ # Go command entry points │ ├── cozystack-api/ │ ├── cozystack-controller/ │ └── cozystack-assets-server/ ├── internal/ # Internal Go packages │ ├── controller/ # Controller implementations │ └── lineagecontrollerwebhook/ ├── pkg/ # Public Go packages │ ├── apis/ │ ├── apiserver/ │ └── registry/ └── api/ # Kubernetes API definitions (CRDs) └── v1alpha1/ ``` ## Package Structure Every package is a Helm chart following the umbrella chart pattern: ``` packages/// ├── Chart.yaml # Chart definition and parameter docs ├── Makefile # Development workflow targets ├── charts/ # Vendored upstream charts ├── images/ # Dockerfiles and image build context ├── patches/ # Optional upstream chart patches ├── templates/ # Additional manifests ├── templates/dashboard-resourcemap.yaml # Dashboard resource mapping ├── values.yaml # Override values for upstream └── values.schema.json # JSON schema for validation and UI ``` ## Build and Development Commands Root targets (run from the repo root): ```bash make build # Build all Docker images (needs: docker, skopeo, jq, gh, helm, yq, GNU tar/sed/awk) make unit-tests # Run unit tests (requires bats-core >= 1.5; GNU parallel is optional) make test-controllers # Run the Go tests under ./internal/... (controllers, contract tests) make generate # Code generation (hack/update-codegen.sh) — CRDs, DeepCopy, clients, RBAC make manifests # Generate CRD manifests and operator YAML variants make cozypkg # Build the cozypkg CLI make test # Full E2E tests (requires a cluster) make prepare-env # Prepare the E2E test environment (apply + prepare-cluster) ``` Per-package targets (run inside `packages/{apps,system,extra,core}//`): ```bash make image # Build the package image make show # Render the Helm template make apply # Apply the Helm release to a cluster (needs NAME, NAMESPACE) make diff # Diff the template against the cluster make test # Package-specific tests make generate # Regenerate values.schema.json + README.md from values.yaml ``` Build environment variables: - `REGISTRY` — Docker registry (default `ghcr.io/cozystack/cozystack`). - `PUSH=1` / `LOAD=0` — control buildx push/load behaviour. - `LOAD=1 PUSH=0` — load images locally instead of pushing. Builds for the host architecture only, because the classic docker image store cannot load a multi-platform index. - `PLATFORM` — the buildx platforms. Defaults to `linux/amd64,linux/arm64` for anything but `LOAD=1`, so a push from either architecture publishes an index both can run. Needs a `docker-container` builder (`BUILDER`) or a docker daemon on the containerd image store, with QEMU registered for the non-native half. Compiling builder stages run natively on `$BUILDPLATFORM` and cross-compile for the target, but every `RUN` step of a stage that runs on the target platform (package installs, `setcap`, install scripts, patch steps) still runs emulated. `packages/core/testing` pins `linux/amd64`, because it carries an x86 KVM sandbox. `packages/core/talos` builds the Talos installer and release assets for the arches in `PLATFORM` with the upstream imager, which needs no QEMU for either; its matchbox image carries the amd64 and arm64 kernel and initramfs whatever `PLATFORM` says, because it network-boots machines of either arch. - `PUSHED_TAGS_LOG` — a file the `image-tags` macro appends every pushed `:` to, one per line. Unset by default; the release build sets it so `hack/stitch-multiarch.sh` can move every tag it pushed, versioned ones included, onto the multi-arch index. The append happens at recipe expansion, so `make -n` writes it too. - `CACHE_TAG` — the tag of the mode=max build cache each image reads (and writes under `WRITE_CACHE=1`). Defaults to `buildcache`. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under `-arm64` with `CACHE_TAG=buildcache-arm64`, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release, `hack/stitch-multiarch.sh` then joins each pair into one index, `hack/verify-multiarch.sh` fails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; see `docs/release.md`. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (see `docs/agents/image-refs.md`). ### Values schema generation Package `values.yaml` files carry annotations (`@param`, `@typedef`, `@field`, `@enum`, `@section`) that `cozyvalues-gen` turns into `values.schema.json` and `README.md`. Run `make generate` in the package after editing annotations. A pre-commit hook runs `make generate` across `packages/apps/*` and `packages/extra/*` on every commit to keep these in sync — see [`contributing.md`](./contributing.md) for the regenerate-before-commit rule. A chart also declares the limits on its own resource name there, with `## @name {string} - ` followed by the constraint tags (`@maxLength`, `@minLength`, `@pattern`) it needs. `cozyvalues-gen` publishes the declaration at the root of `values.schema.json` as `x-cozystack-name`, from where `hack/update-crd.sh` carries it into the ApplicationDefinition, and the aggregated API server enforces it on `metadata.name` for whatever kind the definition describes. The description is the message a rejected name is answered with, so write it as the explanation of the limit. A `@pattern` is unanchored, as JSON Schema specifies, so one meant for the whole name is written `^...$`. A declaration the server cannot enforce as written — a constraint keyword it does not implement, a wrong type, a null, lengths no name satisfies, a pattern that is not RE2 — refuses every create of that kind with a configuration error rather than being dropped. A pattern is checked only to compile, not to match anything, so one that matches no name (`$^`) rejects every name with the pattern as the reason. Annotation keywords such as `title` or `examples`, and `x-` extensions, are ignored. A cap only ever tightens: `prefix + name` must still fit the 53-character Helm release name limit, and a declaration looser than that budget does not widen it. The limits belong in the chart because that is where the facts behind them are — which sub-resources it renders and what release names they take — and a chart that needs nothing beyond the platform-wide limits declares nothing. ## Testing - **Helm and BATS unit tests:** `make helm-unit-tests` runs `hack/helm-unit-tests.sh` over every package that defines a `test` target. A `test` target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as `packages/core/testing` already does with its sandbox flow. `make bats-unit-tests` requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with `pre-commit run bats-unit-tests --hook-stage manual --all-files`; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose `/bin/sh` differs from CI, `make BATS_POSIX_SHELL=dash bats-posix-compat-tests` selects the same interpreter explicitly. `make unit-tests` runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under `./internal/...`; run `make test-controllers` as well. CI invokes both targets together. - **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md). - **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers. ## Conventions ### Helm Charts - Follow **umbrella chart** pattern for system components - Include upstream charts in `charts/` directory (vendored, not referenced) - Override configuration in root `values.yaml` - Use `values.schema.json` for input validation and dashboard UI rendering **No `lookup` that copies another object into the render.** A `lookup` reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one. A `lookup` is fine when a repeated render gives the same output and carries nothing from another object into it: checking that an object exists, keeping a generated Secret's value, keeping an immutable field of the chart's own live object, or a guard that `fail`s on an unsafe change. Keep its empty branch rendering, because offline `helm template` and helm-unittest without a `kubernetesProvider` mock see only that branch; test the other branch with such a mock. To use a fact about the cluster instead, read it from values: `_cluster` and `_namespace` in the `cozystack-values` Secret, or `valuesFrom` on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with `.Capabilities.APIVersions.Has`, since a `lookup` on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object. ### Go Code - Follow standard **Go conventions** and idioms - Use **controller-runtime** patterns for Kubernetes controllers - Prefer **kubebuilder** for API definitions and controllers - Add proper error handling and structured logging ### Git Commits Conventional Commits (`type(scope): description`) with `--signoff`, kept atomic and focused. Full format, scopes, AI-attribution trailer, and PR workflow are in [`contributing.md`](./contributing.md) — the source of truth; do not duplicate them here. ### PackageSource CRD upgrade policy Each component in a `PackageSource` may set `install.upgradeCRDs` to control how CRDs from the chart's `crds/` directory are handled on `HelmRelease` upgrades. Allowed values: `Skip` (default — helm-controller does not touch CRDs on upgrade), `Create` (create new CRDs only), `CreateReplace` (create new and overwrite existing). Set `upgradeCRDs: CreateReplace` for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them. Do **not** set `CreateReplace` blindly: it overwrites every CRD in `crds/` and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually. ### Documentation — where a new document goes Product documentation lives in [`cozystack/website`](https://github.com/cozystack/website), not here. This repository holds only documentation that something in the repository references by path, or that describes the repository's own machinery. Pick the destination before writing, because moving a document later means chasing every `@param`, template comment and test comment that points at it. | Destination | What belongs there | | --- | --- | | `packages///README.md` | Anything scoped to a single package. Prose goes **above** `## Parameters`; that section is `cozyvalues-gen` output and is never hand-edited. The website's `make update-apps` / `update-vms` / `update-networking` / `update-k8s` / `update-services` targets append the whole README onto a published page, but only for the packages named in the `APPS`, `VMS`, `NETWORKING`, `K8S` and `SERVICES` lists of the website `Makefile`, and only from `packages/apps/` and `packages/extra/`: nothing under `packages/system/` is ever imported, and neither is a package that is not on those lists. The sync runs when a release is promoted or tagged, or by hand in the website repository, not when a pull request merges here. Use this route only for a package that is on the lists; for any other the prose reaches no reader, so put it on the website instead. | | `cozystack/website`, under `content/en/docs/` | Operator- and tenant-facing content that spans packages or is task-shaped: guides, how-tos, troubleshooting runbooks, reference tables. Which version directory a page goes into is set by the website's own [`CONTRIBUTING.md`](https://github.com/cozystack/website/blob/main/CONTRIBUTING.md); follow that file rather than a copy of its rules here. | | `docs/` | Documentation that a chart template, migration, script or workflow references by repo-relative path, plus this repository's own process docs (`docs/agents/`, `docs/changelogs/`, `docs/release.md`). | | `api///DESIGN.md`, `pkg/apis//DESIGN.md` | Design of an API group, beside the types it describes. | | [`cozystack/community`](https://github.com/cozystack/community) | Design proposals and RFCs. Not this repository. | When a document could go in either `docs/` or the website, the test is whether a non-documentation file has to link to it by path. `docs/operations/seaweedfs-431-rename-recovery.md` stays because `packages/*/seaweedfs/templates/_naming.tpl` and a platform migration point at it. A guide referenced only from a `values.yaml` `@param` does not qualify — put it on the website and link the URL. Documenting a feature's current limitations is useful and belongs with the feature. Documenting the *development process* around it does not: no "What is NOT in Phase N", no "out of scope for this PR", no "this PR" phrasing, no action plans or review negotiation. A merged document describes what the code does now, and states a limitation as a limitation rather than as a roadmap position. Scope, phasing and follow-ups belong in the pull request description or an issue. A document that names an unmerged pull request or an unreleased dependency needs a tracking issue, because nothing re-checks that claim later and it silently becomes false once the thing ships. ### Domain references for specific tasks Before working on these areas, read the relevant doc first: - **Backup subsystem design:** `api/backups/v1alpha1/DESIGN.md` - **A new app package:** copy the structure of an existing one (e.g. `packages/apps/postgres/`) ## Things Agents Should Not Do ### Never Edit These - Do not modify files in `/vendor/` (Go dependencies) - Do not edit generated files: `zz_generated.*.go` - Do not change `go.mod`/`go.sum` manually (use `go get`) - Do not edit upstream charts in `packages/*/charts/` directly (use patches) - Do not modify image digests in `values.yaml` (generated by build) ### Version Control - Do not commit built artifacts from `_out` - Do not commit test artifacts or temporary files ### Git Operations - Do not force push to main/master - Do not update git config - Do not perform destructive operations without explicit request ### Core Components - Do not modify `packages/core/platform/` without understanding migration impact