Files
cozystack/docs/agents/overview.md
T
Aleksei Sviridkin 1e7ba63c57 docs(agents): define testing and authoring conventions
Require tests for behavior changes without inventing failures for prose
or unchanged upstream updates. Keep new image build filenames explicit
and preserve the existing lookup policy.

Correct the assertion guidance: Bats can fail on a final negation,
while cozytest appends a successful return. Separate numeric validation
from text coercion and account for null parent maps in Helm values.

Validation: markdownlint (MD013 excluded) and git diff --check pass.

Fixes: 5137d98e71 ("test(bats): refuse a negated assertion across every suite")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-12 16:52:11 +03:00

22 KiB
Raw Blame History

Cozystack Project Overview

This document provides detailed information about Cozystack project structure and conventions for AI agents.

About Cozystack

Cozystack is an open-source Kubernetes-based platform and framework for building cloud infrastructure. It provides:

  • Managed Services: Databases, VMs, Kubernetes clusters, object storage, and more
  • Multi-tenancy: Full isolation and self-service for tenants
  • GitOps-driven: FluxCD-based continuous delivery
  • Modular Architecture: Extensible with custom packages and services
  • Developer Experience: Simplified local development with cozyhr tool

The platform exposes infrastructure services via the Kubernetes API with ready-made configs, built-in monitoring, and alerts.

Code Layout

.
├── packages/               # Main directory for cozystack packages
│   ├── core/              # Core platform logic charts (installer, platform)
│   ├── system/            # System charts (CSI, CNI, operators, etc.)
│   ├── apps/              # User-facing charts shown in dashboard catalog
│   └── extra/             # Tenant-specific modules, singleton charts which are used as dependencies
├── dashboards/            # Grafana dashboards for monitoring
├── hack/                  # Helper scripts for local development
│   └── e2e-chainsaw/     # End-to-end application tests (Kyverno Chainsaw)
├── scripts/               # Scripts used by cozystack container
│   └── migrations/       # Version migration scripts
├── docs/                  # Documentation
│   ├── agents/           # AI agent instructions
│   └── changelogs/       # Release changelogs
├── cmd/                   # Go command entry points
│   ├── cozystack-api/
│   ├── cozystack-controller/
│   └── cozystack-assets-server/
├── internal/              # Internal Go packages
│   ├── controller/       # Controller implementations
│   └── lineagecontrollerwebhook/
├── pkg/                   # Public Go packages
│   ├── apis/
│   ├── apiserver/
│   └── registry/
└── api/                   # Kubernetes API definitions (CRDs)
    └── v1alpha1/

Package Structure

Every package is a Helm chart following the umbrella chart pattern:

packages/<category>/<package-name>/
├── Chart.yaml                           # Chart definition and parameter docs
├── Makefile                             # Development workflow targets
├── charts/                              # Vendored upstream charts
├── images/                              # Dockerfiles and image build context
├── patches/                             # Optional upstream chart patches
├── templates/                           # Additional manifests
├── templates/dashboard-resourcemap.yaml # Dashboard resource mapping
├── values.yaml                          # Override values for upstream
└── values.schema.json                   # JSON schema for validation and UI

Build and Development Commands

Root targets (run from the repo root):

make build          # Build all Docker images (needs: docker, skopeo, jq, gh, helm, yq, GNU tar/sed/awk)
make unit-tests     # Run unit tests (requires bats-core >= 1.5; GNU parallel is optional)
make test-controllers # Run the Go tests under ./internal/... (controllers, contract tests)
make generate       # Code generation (hack/update-codegen.sh) — CRDs, DeepCopy, clients, RBAC
make manifests      # Generate CRD manifests and operator YAML variants
make cozypkg        # Build the cozypkg CLI
make test           # Full E2E tests (requires a cluster)
make prepare-env    # Prepare the E2E test environment (apply + prepare-cluster)

Per-package targets (run inside packages/{apps,system,extra,core}/<name>/):

make image          # Build the package image
make show           # Render the Helm template
make apply          # Apply the Helm release to a cluster (needs NAME, NAMESPACE)
make diff           # Diff the template against the cluster
make test           # Package-specific tests
make generate       # Regenerate values.schema.json + README.md from values.yaml

Build environment variables:

  • REGISTRY — Docker registry (default ghcr.io/cozystack/cozystack).
  • PUSH=1 / LOAD=0 — control buildx push/load behaviour.
  • LOAD=1 PUSH=0 — load images locally instead of pushing. Builds for the host architecture only, because the classic docker image store cannot load a multi-platform index.
  • PLATFORM — the buildx platforms. Defaults to linux/amd64,linux/arm64 for anything but LOAD=1, so a push from either architecture publishes an index both can run. Needs a docker-container builder (BUILDER) or a docker daemon on the containerd image store, with QEMU registered for the non-native half. Compiling builder stages run natively on $BUILDPLATFORM and cross-compile for the target, but every RUN step of a stage that runs on the target platform (package installs, setcap, install scripts, patch steps) still runs emulated. packages/core/testing pins linux/amd64, because it carries an x86 KVM sandbox. packages/core/talos builds the Talos installer and release assets for the arches in PLATFORM with the upstream imager, which needs no QEMU for either; its matchbox image carries the amd64 and arm64 kernel and initramfs whatever PLATFORM says, because it network-boots machines of either arch.
  • PUSHED_TAGS_LOG — a file the image-tags macro appends every pushed <repo>:<tag> to, one per line. Unset by default; the release build sets it so hack/stitch-multiarch.sh can move every tag it pushed, versioned ones included, onto the multi-arch index. The append happens at recipe expansion, so make -n writes it too.
  • CACHE_TAG — the tag of the mode=max build cache each image reads (and writes under WRITE_CACHE=1). Defaults to buildcache. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under <tag>-arm64 with CACHE_TAG=buildcache-arm64, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release, hack/stitch-multiarch.sh then joins each pair into one index, hack/verify-multiarch.sh fails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; see docs/release/ci-pipeline.md. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (see docs/agents/image-refs.md).

Values schema generation

Package values.yaml files carry annotations (@param, @typedef, @field, @enum, @section) that cozyvalues-gen turns into values.schema.json and README.md. Run make generate in the package after editing annotations. A pre-commit hook runs make generate across packages/apps/* and packages/extra/* on every commit to keep these in sync — see contributing.md for the regenerate-before-commit rule.

A chart also declares the limits on its own resource name there, with ## @name {string} - <why> followed by the constraint tags (@maxLength, @minLength, @pattern) it needs. cozyvalues-gen publishes the declaration at the root of values.schema.json as x-cozystack-name, from where hack/update-crd.sh carries it into the ApplicationDefinition, and the aggregated API server enforces it on metadata.name for whatever kind the definition describes. The description is the message a rejected name is answered with, so write it as the explanation of the limit. A @pattern is unanchored, as JSON Schema specifies, so one meant for the whole name is written ^...$. A declaration the server cannot enforce as written — a constraint keyword it does not implement, a wrong type, a null, lengths no name satisfies, a pattern that is not RE2 — refuses every create of that kind with a configuration error rather than being dropped. A pattern is checked only to compile, not to match anything, so one that matches no name ($^) rejects every name with the pattern as the reason. Annotation keywords such as title or examples, and x- extensions, are ignored. A cap only ever tightens: prefix + name must still fit the 53-character Helm release name limit, and a declaration looser than that budget does not widen it. The limits belong in the chart because that is where the facts behind them are — which sub-resources it renders and what release names they take — and a chart that needs nothing beyond the platform-wide limits declares nothing.

Testing

  • Helm and BATS unit tests: make helm-unit-tests runs hack/helm-unit-tests.sh over every package that defines a test target. A test target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as packages/core/testing already does with its sandbox flow. make bats-unit-tests requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with pre-commit run bats-unit-tests --hook-stage manual --all-files; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose /bin/sh differs from CI, make BATS_POSIX_SHELL=dash bats-posix-compat-tests selects the same interpreter explicitly. make unit-tests runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under ./internal/...; run make test-controllers as well. CI invokes both targets together.
  • E2E tests: Kyverno Chainsaw suites in hack/e2e-chainsaw/ (one directory per app), run with chainsaw test. Cluster bootstrap (hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in e2e-testing.md.
  • Go tests: standard go test, with Ginkgo/Gomega for controllers.

Test first, failure path first

These rules apply to changes with testable behavior. Documentation-only edits and unmodified upstream updates need the relevant checks, not an artificial failing test. First-party adaptations made during a dependency bump remain covered by these rules.

  1. Write the test before changing the implementation and watch it fail for the intended reason, not on a typo or a missing fixture. Then make it green, then refactor under it.
  2. Cover relevant failure paths before the happy path: empty or null values, invalid input, a number where a string is expected, a missing resource, a timeout or an API error. Assert the resulting message, exit status and post-failure state required by the operation's contract.
  3. Fix regressions in the implementation. Do not skip, weaken or delete a test, or silence a linter, merely to get a green run. When the intended contract changes, update or replace the obsolete assertions and retain coverage of applicable failure cases.
  4. A stub needs a failing test for the intended behavior during development. Finish that behavior and make the test pass before merge; do not make the test accept an unintended not-implemented result just to get green CI.
  5. A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked FIXME(#N). The PR that fixes the bug flips the test to pin the fixed behaviour.

Do not negate a BATS assertion with !: it can silently pass. Bats catches a failing negation only when it is the last command in the test body; hack/cozytest.sh appends a successful return and misses it there too. See §11 of e2e-testing.md.

Linters

Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline.

Markdown follows these markdownlint rules: every code fence names a language (text when nothing else fits), table separators have spaces around the pipes (| --- |), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as Prose Formatting explains.

Conventions

Helm Charts

  • Follow umbrella chart pattern for system components
  • Include upstream charts in charts/ directory (vendored, not referenced)
  • Override configuration in root values.yaml
  • Use values.schema.json for input validation and dashboard UI rendering
  • An unquoted numeric value (name: 123) reaches a template as a number: len on it aborts the render with len of type float64 (int64 for a value given with --set) before any fail message is reached, and if not $v treats 0 as unset. For fields validated as text that permit numeric inputs, reject missing or null values with kindIs "invalid", then coerce with printf "%v" and check emptiness with eq $v "". Coercing null first produces the non-empty string <nil>.
  • For integer fields, keep string inputs on their lexical validation path. Validate numeric inputs for type, integrality and bounds before conversion, then format the validated integer in decimal. Generic printf "%v" can emit exponent notation for a float (4200000000 becomes 4.2e+09), so it is not integer normalization.
  • For nested values, validate or normalize each parent map before reading its children; checking only the leaf cannot prevent a nil-pointer error from a missing or null parent.

No lookup that copies another object into the render. A lookup reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one.

A lookup is fine when a repeated render gives the same output and carries nothing from another object into it: checking that an object exists, keeping a generated Secret's value, keeping an immutable field of the chart's own live object, or a guard that fails on an unsafe change. Keep its empty branch rendering, because offline helm template and helm-unittest without a kubernetesProvider mock see only that branch; test the other branch with such a mock.

To use a fact about the cluster instead, read it from values: _cluster and _namespace in the cozystack-values Secret, or valuesFrom on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with .Capabilities.APIVersions.Has, since a lookup on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object.

Container images

For a new container image built from source maintained in this repository, name the build definition Containerfile and pass it to docker buildx build with --file in the package Makefile. Keep existing Dockerfiles and vendored build definitions unchanged.

Examples and fixtures

Illustrative addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24), IPv6 from RFC 3849 (2001:db8::/32), domains from RFC 2606 and RFC 6761 (example.com, .example, .test, .invalid), ASNs from RFC 5398 (64496–64511, 65536–65551), MAC addresses from RFC 7042 (00:00:5E:00:53:00–00:00:5E:00:53:FF). Where no reserved range exists, use an obviously synthetic placeholder such as <NODE_IP>.

When a test depends on a protocol-defined address's behavior, use that value, such as 127.0.0.1 or localhost for a local test server. Keep network tests isolated; this exception does not permit arbitrary live endpoints.

Go Code

  • Follow standard Go conventions and idioms
  • Use controller-runtime patterns for Kubernetes controllers
  • Prefer kubebuilder for API definitions and controllers
  • Add proper error handling and structured logging

Git Commits

Conventional Commits (type(scope): description) with --signoff, kept atomic and focused. Full format, scopes, AI-attribution trailer, and PR workflow are in contributing.md — the source of truth; do not duplicate them here.

PackageSource CRD upgrade policy

Each component in a PackageSource may set install.upgradeCRDs to control how CRDs from the chart's crds/ directory are handled on HelmRelease upgrades. Allowed values: Skip (default — helm-controller does not touch CRDs on upgrade), Create (create new CRDs only), CreateReplace (create new and overwrite existing).

Set upgradeCRDs: CreateReplace for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them.

Do not set CreateReplace blindly: it overwrites every CRD in crds/ and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually.

Documentation — where a new document goes

Product documentation lives in cozystack/website, not here. This repository holds only documentation that something in the repository references by path, or that describes the repository's own machinery. Pick the destination before writing, because moving a document later means chasing every @param, template comment and test comment that points at it.

Destination What belongs there
packages/<group>/<name>/README.md Anything scoped to a single package. Prose goes above ## Parameters; that section is cozyvalues-gen output and is never hand-edited. The website's make update-apps / update-vms / update-networking / update-k8s / update-services targets append the whole README onto a published page, but only for the packages named in the APPS, VMS, NETWORKING, K8S and SERVICES lists of the website Makefile, and only from packages/apps/ and packages/extra/: nothing under packages/system/ is ever imported, and neither is a package that is not on those lists. The sync runs when a release is promoted or tagged, or by hand in the website repository, not when a pull request merges here. Use this route only for a package that is on the lists; for any other the prose reaches no reader, so put it on the website instead.
cozystack/website, under content/en/docs/ Operator- and tenant-facing content that spans packages or is task-shaped: guides, how-tos, troubleshooting runbooks, reference tables. Which version directory a page goes into is set by the website's own CONTRIBUTING.md; follow that file rather than a copy of its rules here.
docs/ Documentation that a chart template, migration, script or workflow references by repo-relative path, plus this repository's own process docs (docs/agents/, docs/changelogs/, docs/release.md and the pages under docs/release/).
api/<group>/<version>/DESIGN.md, pkg/apis/<group>/DESIGN.md Design of an API group, beside the types it describes.
cozystack/community Design proposals and RFCs. Not this repository.

When a document could go in either docs/ or the website, the test is whether a non-documentation file has to link to it by path. docs/operations/seaweedfs-431-rename-recovery.md stays because packages/*/seaweedfs/templates/_naming.tpl and a platform migration point at it. A guide referenced only from a values.yaml @param does not qualify — put it on the website and link the URL.

Documenting a feature's current limitations is useful and belongs with the feature. Documenting the development process around it does not: no "What is NOT in Phase N", no "out of scope for this PR", no "this PR" phrasing, no action plans or review negotiation. A merged document describes what the code does now, and states a limitation as a limitation rather than as a roadmap position. Scope, phasing and follow-ups belong in the pull request description or an issue.

A document that names an unmerged pull request or an unreleased dependency needs a tracking issue, because nothing re-checks that claim later and it silently becomes false once the thing ships.

Domain references for specific tasks

Before working on these areas, read the relevant doc first:

  • Backup subsystem design: api/backups/v1alpha1/DESIGN.md
  • A new app package: copy the structure of an existing one (e.g. packages/apps/postgres/)

Things Agents Should Not Do

Never Edit These

  • Do not modify files in /vendor/ (Go dependencies)
  • Do not edit generated files: zz_generated.*.go
  • Do not change go.mod/go.sum manually (use go get)
  • Do not edit upstream charts in packages/*/charts/ directly (use patches)
  • Do not modify image digests in values.yaml (generated by build)

Version Control

  • Do not commit built artifacts from _out
  • Do not commit test artifacts or temporary files

Git Operations

  • Do not force push to main/master
  • Do not update git config
  • Do not perform destructive operations without explicit request

Core Components

  • Do not modify packages/core/platform/ without understanding migration impact