<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, security (a fix that comes through a security advisory), docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does <!-- Link the issues this PR addresses. For a complete fix targeting the default branch, use Closes #N (Closes owner/repository#N for another repository). Otherwise use a non-closing reference; release-branch PRs need manual issue linking and closure when appropriate. --> The same mistakes kept coming back in reviews of agent-written changes. A failure was blamed on whatever changed just before it, with no mechanism behind the claim. Commands ran against the current kubectl context while the work moved between the management cluster and tenant clusters. Prose edits got invented failing tests, and behavior changes landed without any. References and review feedback were the other source. Review corrections were stacked as new commits instead of rewriting the change. Closing keywords went into commits for partial fixes, while the PR itself did not link the issue. Feedback in review bodies and conversation comments was missed because only inline threads were read. A rerun of the fork E2E workflow was taken as proof that an unmerged edit to that workflow works. I wrote these down as rules, so a review does not have to teach the same thing again. They go into `AGENTS.md` and `docs/agents/`, which agents and people both read, and the PR template now asks to link the issue a PR addresses. This also fixes my earlier advice on negated Bats assertions: such an assertion can pass silently, but `bats` does fail it when it is the last command of the test. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories The ccp package commit template and the community contributor guide restate parts of these rules. Both follow-ups are open issues, to be done once this lands. <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: [issue #29](https://github.com/cozystack/ccp/issues/29) - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [x] [cozystack/community](https://github.com/cozystack/community) - follow-up: [issue #125](https://github.com/cozystack/community/issues/125) ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note docs(agents): write recurring review findings down as agent rules covering root-cause evidence, explicit cluster targets, tests for behavior changes, commit and issue references, and review feedback outside inline threads. ```
23 KiB
Cozystack Project Overview
This document provides detailed information about Cozystack project structure and conventions for AI agents.
About Cozystack
Cozystack is an open-source Kubernetes-based platform and framework for building cloud infrastructure. It provides:
- Managed Services: Databases, VMs, Kubernetes clusters, object storage, and more
- Multi-tenancy: Full isolation and self-service for tenants
- GitOps-driven: FluxCD-based continuous delivery
- Modular Architecture: Extensible with custom packages and services
- Developer Experience: Simplified local development with cozyhr tool
The platform exposes infrastructure services via the Kubernetes API with ready-made configs, built-in monitoring, and alerts.
Code Layout
.
├── packages/ # Main directory for cozystack packages
│ ├── core/ # Core platform logic charts (installer, platform)
│ ├── system/ # System charts (CSI, CNI, operators, etc.)
│ ├── apps/ # User-facing charts shown in dashboard catalog
│ └── extra/ # Tenant-specific modules, singleton charts which are used as dependencies
├── dashboards/ # Grafana dashboards for monitoring
├── hack/ # Helper scripts for local development
│ └── e2e-chainsaw/ # End-to-end application tests (Kyverno Chainsaw)
├── scripts/ # Scripts used by cozystack container
│ └── migrations/ # Version migration scripts
├── docs/ # Documentation
│ ├── agents/ # AI agent instructions
│ └── changelogs/ # Release changelogs
├── cmd/ # Go command entry points
│ ├── cozystack-api/
│ ├── cozystack-controller/
│ └── cozystack-assets-server/
├── internal/ # Internal Go packages
│ ├── controller/ # Controller implementations
│ └── lineagecontrollerwebhook/
├── pkg/ # Public Go packages
│ ├── apis/
│ ├── apiserver/
│ └── registry/
└── api/ # Kubernetes API definitions (CRDs)
└── v1alpha1/
Package Structure
Every package is a Helm chart following the umbrella chart pattern:
packages/<category>/<package-name>/
├── Chart.yaml # Chart definition and parameter docs
├── Makefile # Development workflow targets
├── charts/ # Vendored upstream charts
├── images/ # Dockerfiles and image build context
├── patches/ # Optional upstream chart patches
├── templates/ # Additional manifests
├── templates/dashboard-resourcemap.yaml # Dashboard resource mapping
├── values.yaml # Override values for upstream
└── values.schema.json # JSON schema for validation and UI
Build and Development Commands
Root targets (run from the repo root):
make build # Build all Docker images (needs: docker, skopeo, jq, gh, helm, yq, GNU tar/sed/awk)
make unit-tests # Run unit tests (requires bats-core >= 1.5; GNU parallel is optional)
make test-controllers # Run the Go tests under ./internal/... (controllers, contract tests)
make generate # Code generation (hack/update-codegen.sh) — CRDs, DeepCopy, clients, RBAC
make manifests # Generate CRD manifests and operator YAML variants
make cozypkg # Build the cozypkg CLI
make test # Full E2E tests (requires a cluster)
make prepare-env # Prepare the E2E test environment (apply + prepare-cluster)
Per-package targets (run inside packages/{apps,system,extra,core}/<name>/):
make image # Build the package image
make show # Render the Helm template
make apply # Apply the Helm release to a cluster (needs NAME, NAMESPACE)
make diff # Diff the template against the cluster
make test # Package-specific tests
make generate # Regenerate values.schema.json + README.md from values.yaml
Build environment variables:
REGISTRY— Docker registry (defaultghcr.io/cozystack/cozystack).PUSH=1/LOAD=0— control buildx push/load behaviour.LOAD=1 PUSH=0— load images locally instead of pushing. Builds for the host architecture only, because the classic docker image store cannot load a multi-platform index.PLATFORM— the buildx platforms. Defaults tolinux/amd64,linux/arm64for anything butLOAD=1, so a push from either architecture publishes an index both can run. Needs adocker-containerbuilder (BUILDER) or a docker daemon on the containerd image store, with QEMU registered for the non-native half. Compiling builder stages run natively on$BUILDPLATFORMand cross-compile for the target, but everyRUNstep of a stage that runs on the target platform (package installs,setcap, install scripts, patch steps) still runs emulated.packages/core/testingpinslinux/amd64, because it carries an x86 KVM sandbox.packages/core/talosbuilds the Talos installer and release assets for the arches inPLATFORMwith the upstream imager, which needs no QEMU for either; its matchbox image carries the amd64 and arm64 kernel and initramfs whateverPLATFORMsays, because it network-boots machines of either arch.PUSHED_TAGS_LOG— a file theimage-tagsmacro appends every pushed<repo>:<tag>to, one per line. Unset by default; the release build sets it sohack/stitch-multiarch.shcan move every tag it pushed, versioned ones included, onto the multi-arch index. The append happens at recipe expansion, somake -nwrites it too.CACHE_TAG— the tag of the mode=max build cache each image reads (and writes underWRITE_CACHE=1). Defaults tobuildcache. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under<tag>-arm64withCACHE_TAG=buildcache-arm64, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release,hack/stitch-multiarch.shthen joins each pair into one index,hack/verify-multiarch.shfails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; seedocs/release/ci-pipeline.md. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (seedocs/agents/image-refs.md).
Values schema generation
Package values.yaml files carry annotations (@param, @typedef, @field, @enum, @section) that cozyvalues-gen turns into values.schema.json and README.md. Run make generate in the package after editing annotations. A pre-commit hook runs make generate across packages/apps/* and packages/extra/* on every commit to keep these in sync — see contributing.md for the regenerate-before-commit rule.
A chart also declares the limits on its own resource name there, with ## @name {string} - <why> followed by the constraint tags (@maxLength, @minLength, @pattern) it needs. cozyvalues-gen publishes the declaration at the root of values.schema.json as x-cozystack-name, from where hack/update-crd.sh carries it into the ApplicationDefinition, and the aggregated API server enforces it on metadata.name for whatever kind the definition describes. The description is the message a rejected name is answered with, so write it as the explanation of the limit. A @pattern is unanchored, as JSON Schema specifies, so one meant for the whole name is written ^...$. A declaration the server cannot enforce as written — a constraint keyword it does not implement, a wrong type, a null, lengths no name satisfies, a pattern that is not RE2 — refuses every create of that kind with a configuration error rather than being dropped. A pattern is checked only to compile, not to match anything, so one that matches no name ($^) rejects every name with the pattern as the reason. Annotation keywords such as title or examples, and x- extensions, are ignored. A cap only ever tightens: prefix + name must still fit the 53-character Helm release name limit, and a declaration looser than that budget does not widen it. The limits belong in the chart because that is where the facts behind them are — which sub-resources it renders and what release names they take — and a chart that needs nothing beyond the platform-wide limits declares nothing.
Testing
- Helm and BATS unit tests:
make helm-unit-testsrunshack/helm-unit-tests.shover every package that defines atesttarget. Atesttarget is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, aspackages/core/testingalready does with its sandbox flow.make bats-unit-testsrequires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass withpre-commit run bats-unit-tests --hook-stage manual --all-files; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose/bin/shdiffers from CI,make BATS_POSIX_SHELL=dash bats-posix-compat-testsselects the same interpreter explicitly.make unit-testsruns the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under./internal/...; runmake test-controllersas well. CI invokes both targets together. The backport-batch tests are not part of it: runmake test-backport-batchafter changing anything listed incmd/backport-batch/test-inputs.txt, which is also what CI keys them on. - E2E tests: Kyverno Chainsaw suites in
hack/e2e-chainsaw/(one directory per app), run withchainsaw test. Cluster bootstrap (hack/e2e-install-cozystack.bats) and the OpenAPI checks (hack/e2e-test-openapi.bats) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live ine2e-testing.md. - Go tests: standard
go test, with Ginkgo/Gomega for controllers.
Test first, failure path first
These rules apply to changes with testable behavior. Documentation-only edits and unmodified upstream updates need the relevant checks, not an artificial failing test. First-party adaptations made during a dependency bump remain covered by these rules.
- Write the test before changing the implementation and watch it fail for the intended reason, not on a typo or a missing fixture. Then make it green, then refactor under it.
- Cover relevant failure paths before the happy path: empty or null values, invalid input, a number where a string is expected, a missing resource, a timeout or an API error. Assert the resulting message, exit status and post-failure state required by the operation's contract.
- Fix regressions in the implementation. Do not skip, weaken or delete a test, or silence a linter, merely to get a green run. When the intended contract changes, update or replace the obsolete assertions and retain coverage of applicable failure cases.
- A stub needs a failing test for the intended behavior during development. Finish that behavior and make the test pass before merge; do not make the test accept an unintended not-implemented result just to get green CI.
- A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked
FIXME(#N). The PR that fixes the bug flips the test to pin the fixed behaviour.
Do not negate a BATS assertion with !: it can silently pass. Bats catches a failing negation only when it is the last command in the test body; hack/cozytest.sh appends a successful return and misses it there too. See §11 of e2e-testing.md.
Linters
Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline.
Markdown follows these markdownlint rules: every code fence names a language (text when nothing else fits), table separators have spaces around the pipes (| --- |), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as Prose Formatting explains.
Conventions
Helm Charts
- Follow umbrella chart pattern for system components
- Include upstream charts in
charts/directory (vendored, not referenced) - Override configuration in root
values.yaml - Use
values.schema.jsonfor input validation and dashboard UI rendering - An unquoted numeric value (
name: 123) reaches a template as a number:lenon it aborts the render withlen of type float64(int64for a value given with--set) before anyfailmessage is reached, andif not $vtreats0as unset. For fields validated as text that permit numeric inputs, reject missing or null values withkindIs "invalid", then coerce withprintf "%v"and check emptiness witheq $v "". Coercing null first produces the non-empty string<nil>. - For integer fields, keep string inputs on their lexical validation path. Validate numeric inputs for type, integrality and bounds before conversion, then format the validated integer in decimal. Generic
printf "%v"can emit exponent notation for a float (4200000000becomes4.2e+09), so it is not integer normalization. - For nested values, validate or normalize each parent map before reading its children; checking only the leaf cannot prevent a nil-pointer error from a missing or null parent.
No lookup that copies another object into the render. A lookup reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one.
A lookup is fine when a repeated render gives the same output and carries nothing from another object into it: checking that an object exists, keeping a generated Secret's value, keeping an immutable field of the chart's own live object, or a guard that fails on an unsafe change. Keep its empty branch rendering, because offline helm template and helm-unittest without a kubernetesProvider mock see only that branch; test the other branch with such a mock.
To use a fact about the cluster instead, read it from values: _cluster and _namespace in the cozystack-values Secret, or valuesFrom on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with .Capabilities.APIVersions.Has, since a lookup on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object.
Container images
For a new container image built from source maintained in this repository, name the build definition Containerfile and pass it to docker buildx build with --file in the package Makefile. Keep existing Dockerfiles and vendored build definitions unchanged.
Examples and fixtures
Illustrative addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24), IPv6 from RFC 3849 (2001:db8::/32), domains from RFC 2606 and RFC 6761 (example.com, .example, .test, .invalid), ASNs from RFC 5398 (64496–64511, 65536–65551), MAC addresses from RFC 7042 (00:00:5E:00:53:00–00:00:5E:00:53:FF). Where no reserved range exists, use an obviously synthetic placeholder such as <NODE_IP>.
When a test depends on a protocol-defined address's behavior, use that value, such as 127.0.0.1 or localhost for a local test server. Keep network tests isolated; this exception does not permit arbitrary live endpoints.
Go Code
- Follow standard Go conventions and idioms
- Use controller-runtime patterns for Kubernetes controllers
- Prefer kubebuilder for API definitions and controllers
- Add proper error handling and structured logging
Git Commits
Conventional Commits (type(scope): description) with --signoff, kept atomic and focused. Full format, scopes, AI-attribution trailer, and PR workflow are in contributing.md — the source of truth; do not duplicate them here.
PackageSource CRD upgrade policy
Each component in a PackageSource may set install.upgradeCRDs to control how CRDs from the chart's crds/ directory are handled on HelmRelease upgrades. Allowed values: Skip (default — helm-controller does not touch CRDs on upgrade), Create (create new CRDs only), CreateReplace (create new and overwrite existing).
Set upgradeCRDs: CreateReplace for operators whose upstream regularly adds new CRDs between versions (etcd-operator, cnpg, kubevirt, kamaji). Without it, new CRDs from a chart bump do not land on existing clusters — only fresh installs get them.
Do not set CreateReplace blindly: it overwrites every CRD in crds/ and can cause silent data loss if upstream drops a field from a CRD that has live objects. Only enable it for operators whose schema evolution is additive-only. When in doubt, leave it unset and apply new CRDs manually.
Documentation — where a new document goes
Product documentation lives in cozystack/website, not here. This repository holds only documentation that something in the repository references by path, or that describes the repository's own machinery. Pick the destination before writing, because moving a document later means chasing every @param, template comment and test comment that points at it.
| Destination | What belongs there |
|---|---|
packages/<group>/<name>/README.md |
Anything scoped to a single package. Prose goes above ## Parameters; that section is cozyvalues-gen output and is never hand-edited. The website's make update-apps / update-vms / update-networking / update-k8s / update-services targets append the whole README onto a published page, but only for the packages named in the APPS, VMS, NETWORKING, K8S and SERVICES lists of the website Makefile, and only from packages/apps/ and packages/extra/: nothing under packages/system/ is ever imported, and neither is a package that is not on those lists. The sync runs when a release is promoted or tagged, or by hand in the website repository, not when a pull request merges here. Use this route only for a package that is on the lists; for any other the prose reaches no reader, so put it on the website instead. |
cozystack/website, under content/en/docs/ |
Operator- and tenant-facing content that spans packages or is task-shaped: guides, how-tos, troubleshooting runbooks, reference tables. Which version directory a page goes into is set by the website's own CONTRIBUTING.md; follow that file rather than a copy of its rules here. |
docs/ |
Documentation that a chart template, migration, script or workflow references by repo-relative path, plus this repository's own process docs (docs/agents/, docs/changelogs/, docs/release.md and the pages under docs/release/). |
api/<group>/<version>/DESIGN.md, pkg/apis/<group>/DESIGN.md |
Design of an API group, beside the types it describes. |
cozystack/community |
Design proposals and RFCs. Not this repository. |
When a document could go in either docs/ or the website, the test is whether a non-documentation file has to link to it by path. docs/operations/seaweedfs-431-rename-recovery.md stays because packages/*/seaweedfs/templates/_naming.tpl and a platform migration point at it. A guide referenced only from a values.yaml @param does not qualify — put it on the website and link the URL.
Documenting a feature's current limitations is useful and belongs with the feature. Documenting the development process around it does not: no "What is NOT in Phase N", no "out of scope for this PR", no "this PR" phrasing, no action plans or review negotiation. A merged document describes what the code does now, and states a limitation as a limitation rather than as a roadmap position. Scope, phasing and follow-ups belong in the pull request description or an issue.
An app's README and the field descriptions in its values.yaml describe what the app does now, for the person using it: no history ("moves from", "previously", "used to", rollout phase names) and no account of how another chart behaved before. A change users must act on is announced by a ! in the pull request title and described on the website's per-version upgrade page (content/en/docs/next/operations/cluster/upgrade-to-vX.Y.md), so an app README carries no history or breaking-changes section. A description says what the field does, what it accepts and what it refuses, not what it replaced, and does not tell the reader to keep it in sync with another chart; when two charts have to agree, a test says so. A change that touches a README or a description fixes what is wrong in that place while it is there.
A document that names an unmerged pull request or an unreleased dependency needs a tracking issue, because nothing re-checks that claim later and it silently becomes false once the thing ships.
Domain references for specific tasks
Before working on these areas, read the relevant doc first:
- Backup subsystem design:
api/backups/v1alpha1/DESIGN.md - A new app package: copy the structure of an existing one (e.g.
packages/apps/postgres/)
Things Agents Should Not Do
Never Edit These
- Do not modify files in
/vendor/(Go dependencies) - Do not edit generated files:
zz_generated.*.go - Do not change
go.mod/go.summanually (usego get) - Do not edit upstream charts in
packages/*/charts/directly (use patches) - Do not modify image digests in
values.yaml(generated by build)
Version Control
- Do not commit built artifacts from
_out - Do not commit test artifacts or temporary files
Git Operations
- Do not force push to main/master
- Do not update git config
- Do not perform destructive operations without explicit request
Core Components
- Do not modify
packages/core/platform/without understanding migration impact