Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 47638f8569)
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 2b1d3e904d)
## What this PR does
Backport of #3121 to `release-1.5`.
Makes node group `resources` optional, so a node group sized purely by
`instanceType` (with `resources` omitted) passes `values.schema.json`
validation and installs. When a group sets both `instanceType` and
explicit `resources` (both `cpu` and `memory`), the explicit resources
take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt does not
reject a VM that references an instancetype while overriding
`domain.cpu`/`memory`. A partial spec (only one of `cpu`/`memory`)
alongside an `instanceType` fails fast at render time.
Cherry-picked from `a349c753`; conflicts resolved against the
`release-1.5` tree (the management-cluster Talos worker bootstrap stack
present on `main` is not on this branch, so the schema/template/test
layout differ). Generated files (`values.schema.json`, `README.md`,
cozyrds, API types) regenerated with `make generate` to match. `helm
unittest` passes (134 tests), including the four new node-group sizing
cases adapted to this branch's document layout.
### Release note
```release-note
fix(kubernetes): make node group resources optional when instanceType is set
```
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
(cherry picked from commit f8e4222b7d)
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.
Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:
- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
external access via `service.type: LoadBalancer` and keep `primaryService`
LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
(no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
-primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
installs are visible in the dashboard.
Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
(cherry picked from commit c7e8f4d975)
The tenant etcd, ingress and monitoring HelmReleases create VM* custom
resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert,
VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system
child of monitoring) guarded by the victoria-metrics-operator validating
webhook with failurePolicy: Fail. On a cold install the operator pod can
be briefly unavailable on :9443, so these releases reconcile before the
webhook backend is serving and are rejected with connection refused. They
carry install.remediation.retries: -1 and would eventually recover, but
the failed-install -> rollback -> retry churn can outlast a cold-install
readiness deadline and surfaces as spurious install failures.
Add a Flux spec.dependsOn from each of these releases to the
victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator,
so they wait for the operator to be Ready before creating VM* resources.
The system monitoring components gate on the operator the same way, via a
variant-level dependsOn. The cross-shard reference (tenant releases are
sharded, the operator release is not) resolves via helm-controller's
uncached dependency check (APIReader, v1.1.0+).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 6db5ae9c6f)
A node group is sized either by instanceType or by explicit resources, but the
generated values schema required resources on every node group. A node group
with instanceType and no resources — or an empty resources dropped during
typed-CR to HelmRelease translation — failed schema validation with "missing
property 'resources'" even though sizing was fully specified.
Mark resources optional in the node group definition and regenerate the schema,
API types, README, and ApplicationDefinition. The chart template emits
domain.cpu/memory only when resources is present, so a node group sized purely
by instanceType now passes validation and installs.
When a node group sets both instanceType and explicit resources, explicit
resources take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit
domain.cpu/memory with no instancetype to conflict with. instanceType stays
required and defaulted (no schema change), so the console's required
instanceType dropdown keeps working — sending the default instanceType plus
explicit resources simply sizes the VM by resources. This replaces an earlier
render-time hard failure on the both-set case, which was unreachable to clear
from the dashboard (the required dropdown cannot be emptied).
Explicit resources are all-or-nothing alongside an instanceType: a partial spec
(only cpu, or only memory) on a group that carries a non-empty instanceType
would emit one domain override next to the instancetype — which KubeVirt rejects
— so it now fails at render time with a clear error. Partial resources on a
group without an instanceType (used to tune kubelet reservations) are unaffected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit a349c753ad)
The seaweedfs COSI driver shipped at v0.1.2 hardcodes read-write S3
actions (Read, Write, List, Tagging) for every BucketAccess and never
reads the accessPolicy parameter. The chart already defines a -readonly
BucketAccessClass carrying parameters.accessPolicy: readonly, but v0.1.2
ignores it, so a credential meant to be read-only can still write
(upload and delete) objects.
Bump the driver to v0.3.1, which honours accessPolicy: readonly by
issuing Read,List-only credentials (read-write and the empty default
stay Read,Write,List,Tagging). The override lives in the package
values.yaml rather than the vendored chart so it survives make update;
the sister extra/seaweedfs chart already runs v0.3.1.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 0c6426feaa)
The cozy-harbor wrapper never passed storageClass to the jobservice
jobLog PVC, so it was created without a storageClassName and the
default-StorageClass admission stamped the cluster default. Because PVC
storageClassName is immutable, this caused rendered-vs-live drift on
upgrade. Preserve an existing PVC's class via lookup and fall back to the
configured storageClass for fresh installs.
The trivy StatefulSet's volumeClaimTemplate had the same latent drift --
.Values.storageClass was forwarded blindly into a VCT that cannot be
patched at all -- so it gets the same lookup-preserve treatment, keyed
off data-<release>-trivy-0.
A live object pinned to an explicit "" (no dynamic provisioning) now
round-trips through upstream's "-" sentinel instead of collapsing to an
omitted storageClassName, and the spec lookup is guarded with
`| default dict` so a PVC carrying no storageClassName key stays omitted.
Adds a helm-unittest regression suite and wires a test target so
make unit-tests stops skipping packages/apps/harbor.
Fixes#2368
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit bdf520bf74)
The s3-service-name.patch renamed the S3 Service to <name>-s3
(seaweedfs-s3), but several consumers still resolved it through
componentName -> <fullname>-s3. name and fullname only coincide when the
release name equals the chart name; the shipped release is
seaweedfs-system, so they pointed at seaweedfs-system-s3 while the real
service stayed seaweedfs-s3:
- the S3 ingress backend and the iceberg ingress backend, which left
ingress-nginx returning 503 on the external S3 endpoint.
The COSI provisioner in-cluster ENDPOINT was wrong on the same axis and
also on the port. Both the s3-gateway branch and the filer-S3 branch
rendered a portless URL that defaults to 443, while the s3 service only
listens on the S3 port (8333); the filer branch additionally targeted
the filer service, which serves no S3. Both in-cluster branches now
resolve seaweedfs-s3 on the configured S3 port.
The default path (s3.ingress.enabled=true) is repaired by the ingress
fix; the COSI in-cluster paths are latent until an operator disables the
ingress. Resolve every consumer through the same name helper the Service
uses, and pin the contract with unittests rendered under the production
release name so the mismatch cannot recur.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 8a14de9abf)
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit ee853b00bd)
entityOperator.topicOperator/userOperator carried no resources, so in tenant
namespaces (the LimitRange defaults containers to 128Mi) the JVM topic-operator
and user-operator are OOMKilled on startup. The entity-operator stays in
CrashLoopBackOff, KafkaTopics/KafkaUsers never reconcile, and on teardown topics
get stuck on the strimzi.io/topic-operator finalizer. Set explicit requests/limits
(256Mi/512Mi) so they no longer inherit the namespace default; add a template test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 1ab5f17cac)
TEMPORARY revert. Restoring persistent EFI/TPM state (4cc60e3) makes
windows.11/2k22/2k25 VMs get a backend-storage PVC that is ReadWriteOnce on the
default StorageClass, which pins the VM to its node and blocks live-migration and
node drains (with evictionStrategy: LiveMigrate this can stall cluster upgrades).
See #3005.
Strip persistent EFI/TPM from the preferences again, but with `yq del` so an
emptied preferredTPM stays `{}` (the v1.8 CRD rejects null) — the bug the original
sed strip caused. Secure Boot and the vTPM stay present, just non-persistent.
Stopgap until vm-state can request an RWX backend (RWX-Block via the upstream
KubeVirt change, or an RWX-Filesystem vmStateStorageClass).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 6ac2601a68)
Merge the AI-generated v1.5.0 changelog with a fuller pass:
- Add an "Upgrade Notes and Required Actions" section (K8s 1.33+, removal
of upgrade.force, GPU permittedHostDevices ownership, MetalLB FRR-K8s /
HTTPS metrics, automatic TLS on externally-published services).
- Expand "Platform Components" with per-component upstream highlights and
CVE references (Flux v2.8 CVE-2026-23990, MetalLB v0.16 CVE-2025-22874),
plus the new frr-k8s subchart and kuberture package.
- Scope the storageClass-immutability entry (#2639) to UI-only enforcement,
with apiserver enforcement tracked in #2657.
- Broaden Development/Testing/CI coverage and add the v1.4.x website docs
backports (#565, #566, #567, #580).
- Correct the loop-device LVM-filter attribution to talm#215 (the prior
ansible-cozystack#51 reference was a dependency bump, not this change).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `Release E2E` workflow's build job logged into `iad.ocir.io` using
`secrets.OCI_REGISTRY_USERNAME` / `secrets.OCI_REGISTRY_PASSWORD` —
secret names that don't exist in the repo. `docker/login-action`
therefore received empty credentials and failed at the very first step
with `Username and password required`, before any image was built or any
test ran.
The workflow has **never** succeeded since it was added (2026-05-02):
all three tag runs to date died at this step — `v1.5.0-rc.2`,
`v1.5.0-rc.3`, and `v1.5.0`.
This switches the login to the configured `OCIR_USER` / `OCIR_TOKEN`
secrets — the same ones `build-main.yaml` and `pull-requests.yaml` use
against the same registry — and adds the per-job Docker config isolation
those workflows already use (`Set up Docker config` step +
`DOCKER_CONFIG` env on the login and build steps), so the login doesn't
clobber the shared `~/.docker/config.json` on the self-hosted runner.
### Release note
```release-note
ci(tests): fix OCIR registry login in the Release E2E workflow so it can build and run the full suite on release tags
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Chores**
* Improved the release workflow with enhanced Docker credential
isolation and configuration handling during the build process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Release E2E build job referenced secrets OCI_REGISTRY_USERNAME /
OCI_REGISTRY_PASSWORD, which do not exist in the repo, so
docker/login-action received empty credentials and failed every run
with "Username and password required". The workflow has never built
since it was added (2026-05-02): v1.5.0-rc.2, v1.5.0-rc.3 and v1.5.0
all died at this step before any image was built or test run.
Use the configured OCIR_USER / OCIR_TOKEN secrets, matching
build-main.yaml and pull-requests.yaml, and isolate the Docker config
per job (Set up Docker config step + DOCKER_CONFIG on the login and
build steps) so login does not clobber the shared ~/.docker/config.json
on the self-hosted runner.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Implements the tenant Flux sharding design from #2817: a new
`flux-shard-operator` system package that spreads tenant HelmReleases
across multiple helm-controller shards, so a noisy tenant (e.g. a
HelmRelease stuck in infinite remediation) cannot degrade the others.
- **Placement controller** owns the tenant→shard assignment: the unit of
placement is the tenant (all of its HelmReleases share one shard label),
distribution is greedy least-loaded weighted by HelmRelease count, with
pinned tenants, threshold-gated rebalance, paced moves, and never moving
deleting tenants. Watches are metadata-only, so the controller does not
decode the helm-controller status-patch firehose. Assignments are
recorded on tenant namespaces; HelmRelease labels remain the source of
truth on restarts.
- **Shard runtime** is provisioned by the operator: `shardCount`
helm-controller Deployments cloned from the flux-aio `flux` Deployment
and sanitised, so the image and feature-gates stay version-synced
automatically. flux-aio itself is unchanged and keeps owning everything
without a shard key.
- **CREATE-only mutating webhook** (`failurePolicy: Ignore`) stamps the
tenant's shard onto every HelmRelease at admission, so each one is born
on the correct shard regardless of creation path; any miss degrades
gracefully to the placement controller.
- The hand-rolled `flux-tenants` Deployment is retired: the operator
drains it (backfill `tenants` → `shard<i>`) and deletes it once empty;
migration 44 removes a leftover only under the same guard.
Ships with `shardCount: auto` (the default): the operator sizes the
shard count from the tenant HelmRelease count — `K = clamp(ceil(H/100),
1, min(16, T))` with hysteresis (scale up past 120 HR/shard, down under
60) — so small clusters stay at one shard (≈ today's behaviour) and
large fleets shard out automatically. An integer value pins the count
explicitly.
One deviation from the design doc: the webhook/controller scope is the
`sharding.fluxcd.io/key` label rather than
`internal.cozystack.io/tenantmodule` — every tenant HelmRelease is born
with `key=tenants` (stamped by ApplicationDefinition `release.labels`
and chart templates), so the webhook rewrites it and the placement
controller continuously relabels stragglers; `tenantmodule` covers only
the 6 tenant chart modules.
### Screenshots
Not a UI change.
### Release note
```release-note
feat(fluxcd): tenant HelmReleases are now reconciled by operator-managed helm-controller shards; the flux-shard-operator balances tenants across `shardCount` instances and retires the legacy flux-tenants deployment
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced **flux-shard-operator** to shard helm-controller workloads
with deterministic tenant placement, optional rebalance controls, and
autosized shard count.
* Added a **CREATE-only mutating webhook** to stamp the correct shard
label on tenant HelmReleases.
* **Bug Fixes**
* Application updates now preserve existing **flux-shard** label
assignments to prevent unwanted shard re-stamping.
* **Chores**
* Added platform packaging for the new component, including an image
build step.
* Removed the legacy tenants deployment and added a migration to retire
it safely.
* **Tests**
* Expanded unit and chart tests for placement, provisioning, webhook
behavior, and configuration parsing.
* **Documentation**
* Added operator documentation describing configuration and metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `Build cache (main)` workflow (`build-main.yaml`, added in #2948)
failed on
its first run:
https://github.com/cozystack/cozystack/actions/runs/27929277423
The `Set up Buildx` step ran without `DOCKER_CONFIG`, so
`docker/setup-buildx-action` created the docker-container builder under
the
runner's default `~/.docker`. The build step runs with
`DOCKER_CONFIG=$RUNNER_TEMP/.docker` (seeded earlier, holding the OCIR
auth), so
buildx couldn't find the builder and failed immediately:
```
ERROR: no builder "builder-6fb46242-…" found
```
The post-job `docker buildx rm` still succeeded — because the action's
cleanup
runs under the default config where the builder actually lived — which
confirms
the builder existed but in the wrong config store.
This sets `DOCKER_CONFIG` on the `Set up Buildx` step so the builder
instance is
created in the same config the build step reads.
Impact: only the main cache-warmer was broken. PR builds `--cache-from`
this
ref, so a missing cache is a cold build, not a failure — but the
warm-cache
speedup from #2948 wasn't being delivered until this fix.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved build pipeline stability by ensuring consistent Docker
configuration across workflow steps. This enhances the reliability of
the container build and deployment process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
During node reprovisioning in tenant Kubernetes clusters, the VMI (and
its hot-plug pod) can outlive the parent VirtualMachine. The upstream
kubevirt-csi-driver treats a missing VM as "detach succeeded", so the
hot-plug pod stays alive and keeps the infra storage device exclusively
attached to the source host — every subsequent attach of that volume
fails, on DRBD with `failed to set source device readwrite`. This was
hit repeatedly in production tenant clusters (see
kubevirt/csi-driver#182).
This PR:
- updates the pinned `kubevirt.io/csi-driver` module to the latest
upstream commit;
- backports kubevirt/csi-driver#184 into the controller wrapper: after
the upstream unpublish returns success, the wrapper re-checks
`VMI.status.volumeStatus` and, if the hot-plug is still reported there
(e.g. because the parent VM is already gone), detaches it from the VMI
directly via the `removevolume` subresource and waits until it
disappears.
The backport is self-contained in the wrapper and can be dropped once
kubevirt/csi-driver#184 merges upstream and the module pin is bumped
past it.
Credit to @mattia-eleuteri for the root-cause analysis and the upstream
fix.
### Screenshots
Not a UI change.
### Release note
```release-note
fix(csi): tenant CSI driver no longer leaves orphan hot-plug volumes exclusively attached to the source host when the VMI outlives its deleted VirtualMachine during node reprovisioning
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced hotplug volume unpublishing with automatic verification to
ensure volumes are fully detached before completion, including retry
logic with up to 2-minute timeout.
* **Dependencies**
* Updated KubeVirt API and CSI driver dependencies to latest versions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The **Auto Patch Release** cron (`.github/workflows/auto-release.yaml`)
walked
*every* `release-X.Y` branch — 21 of them, back to `release-0.9` — and
cut a
`vX.Y.(Z+1)` tag whenever a branch had commits after its latest
published
release. This fires patch releases on long-EOL lines (e.g. `v1.3.6`),
whose
release PRs then sit blocked on unrelated CI failures and accumulate
stray
draft releases, tags and changelog PRs.
This restricts the cron to the **2 newest minor lines**
(`SUPPORTED_LINE_COUNT`).
The window is derived from the live branch list, so it slides
automatically as
new `release-X.Y` branches are cut — once `release-1.5` exists the
window
becomes `{1.5, 1.4}` and the trailing line retires with no further
edits.
Currently active: `release-1.4`, `release-1.3`. Skipped: 19 EOL lines.
### Release note
```release-note
ci(release): the automated patch-release job now only cuts releases for the two newest release branches; older EOL lines no longer receive automatic patch tags.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the automated release workflow to focus patch automation on
only the two newest minor `release-X.Y` maintenance lines.
* Enhanced workflow logging to report the selected branches and any
older/EOL branches skipped.
* **Documentation**
* Refined release documentation to explain the sliding-window selection
for automated patch tagging, including how next patch versions are
determined and when tags are created.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Raises the OpenSSF Scorecard **SAST** check (was **0/10**) by adding
GitHub's recommended CodeQL static analysis for Go. CodeQL is free for
public repositories and is the SAST tool Scorecard recognizes directly.
- Triggers on push to `main`, PRs to `main`, and a weekly schedule (Mon
03:27 UTC, offset from the other Monday cron jobs).
- Results are uploaded to the code-scanning dashboard via
`github/codeql-action/analyze`.
### Why `autobuild`
The Go analysis uses **`build-mode: autobuild`**. Go's CodeQL extractor
only supports `autobuild` or `manual` (not `none`), so autobuild is the
low-friction choice. It discovers every `go.mod` in the tree and
extracts each module separately — the root module, `api/apps/v1alpha1`,
and the three nested image modules a root `go build ./...` never reaches
— giving broader coverage than a single root build. Before falling back
to the Go build it first attempts `make` from the repo root, which runs
the default Makefile goal (including a network `git fetch upstream
--tags`). `setup-go` pins the toolchain to the `go.mod` version so the
extractor uses the project's Go rather than whatever the runner ships.
### Security posture
- Top-level token is **read-only**; the `analyze` job opts into
`security-events: write` (plus `contents: read`, `actions: read`) only.
- `actions/checkout` is pinned to **v6.0.2** (matching `scorecard.yml`)
with `persist-credentials: false`.
- `codeql-action/init` and `/analyze` are pinned to the **v4.36.2 SHA**
— the same commit as `scorecard.yml`'s `upload-sarif`, keeping every
codeql-action ref in the repo on one commit and clear of the v3
deprecation warning.
### Verification
- `actionlint`: **exit 0**.
- Valid YAML.
- No `run:` steps → no untrusted-input injection surface.
> SAST is a ramping check: Scorecard credits the CodeQL workflow's
presence immediately and the score climbs toward full as subsequent
merged PRs get analyzed.
### Release note
```release-note
ci: add CodeQL static analysis (SAST) for Go, running on push/PR to main and weekly
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Enabled automated security scanning via CodeQL on pushes, pull
requests targeting the main branch, and on a weekly schedule.
* Added workflow controls for concurrency and longer execution time,
with support for Go-based analysis using an autobuild setup.
* Updated the SARIF upload step to use the latest CodeQL action version
to improve reliability while maintaining secure, minimal permissions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Promotes universal project guidance that previously lived only in a
local
(gitignored) `CLAUDE.md` into the committed agent docs, so every
contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
- `overview.md`: Build & Development Commands (root + per-package make
targets,
build env vars), Values schema generation, a Testing section, Domain
references
- `contributing.md`: a "Verify Before Considering Done" matrix + Review
Expectations
- Dedup: `overview.md`'s Git Commits block now points to
`contributing.md`
> Note: `overview.md` links to `docs/agents/e2e-testing.md` added in
#2932 — merge
> that one first or together.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Expanded contribution guidance with a “Verify Before Considering Done”
pre-submission checklist and clearer expectations for including
behavioral regression tests in bug-fix PRs.
* Broadened build/development documentation with a “Build and
Development Commands” guide, including required environment variables,
values-schema generation, and test entry points.
* Updated contribution workflow notes by consolidating commit-convention
guidance and adding pointers to task-specific design references.
* Added a Claude entry-point document that directs to the main agents
guide and explains local override placement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `docs/agents/e2e-testing.md` — a routed agent doc capturing the
e2e/CI
stabilization conventions built up over the last few months:
- No retries on deterministic CI steps (only pure-infra `Prepare
environment` retries)
- Event-driven backstops instead of fixed `sleep`; no `EXIT`/`RETURN`
traps
- Fail-fast HelmRelease-readiness gating + the `status.history`
remediation guard
- First-class failure diagnostics and a reviewer checklist
- Test-Impact Analysis (TIA) and its real scope: it trims only the
per-app loop,
runs after install, and most PRs legitimately escalate to the full suite
Wired into `AGENTS.md` so every agent reads it before e2e/CI work.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Introduced comprehensive E2E testing conventions document with
standards for writing, modifying, and reviewing end-to-end tests and CI
processes.
* Documented best practices for test reliability, including guidelines
on retry behavior, event-driven test validation, and cleanup discipline.
* Updated process documentation to reference new E2E testing
conventions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Why
Follow-up to #2509. PR #2509 exposed five HelmRelease generation knobs
as `cozystack-operator` flags
(`--helmrelease-{interval,retry-interval,install-timeout,upgrade-timeout,max-history}`).
The other HelmRelease-generating path — `cozystack-api`'s
`convertApplicationToHelmRelease` in
`pkg/registry/apps/application/rest.go` — was deliberately left out of
#2509's scope and was flagged in review:
> `pkg/registry/apps/application/rest.go` (the other
HelmRelease-generating path) still hardcodes `Interval: 5m`,
`Remediation{Retries:-1}`, no `Strategy`. Out of this PR's scope; worth
a Followups bullet so the rationale doesn't drift across paths.
So today the api-side path hardcodes:
- `Spec.Interval = 5m`
- `Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`
- no `Strategy`
- no `MaxHistory`
Only `HelmInstallTimeout` (per-Application annotation override) flows
through `ReleaseConfig`. This PR brings the api-side to feature parity
with the operator: same flag names, same defaults, same validation, same
generated HelmRelease shape.
## What
### New cozystack-api flags
Names, types, defaults, and validation match the operator's flags
introduced in #2509 verbatim, so both binaries can be tuned with the
same vocabulary.
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
Validated in `CozyServerOptions.Complete()` via
`config.ParsePositiveDuration` so a misconfigured server restarts loudly
instead of waiting until the first Application is created. `MaxHistory <
0` is rejected; `0` is honored (unlimited per Helm semantics, matching
the operator).
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (with `RetryInterval` from the new flag) instead of
`Install/Upgrade.Remediation{Retries: -1}`. Functionally equivalent for
this path (which never relied on remediation firing), but **decouples
failed-install retry timing from `Spec.Interval`** the same way the
operator change did.
`Install.Remediation` / `Upgrade.Remediation` stay nil: retries run
through the single `Strategy.RetryInterval` path, matching
cozystack-operator's `PackageReconciler`. Reintroducing
`Remediation{Retries:-1}` "for safety" would add a second, conflicting
retry mechanism and break parity with the operator path. A regression
test guards this. (The deployed helm-controller CRD has no rule
forbidding a `Strategy` alongside a `Remediation` entry — this is a
parity/clarity guard, not a CRD-validation one.)
### Per-Application timeout override is preserved (and extended)
The `release.cozystack.io/helm-install-timeout` annotation on
`ApplicationDefinition` (`HelmInstallTimeout` in `ReleaseConfig`) still
wins over the new global `HelmReleaseInstallTimeout` /
`HelmReleaseUpgradeTimeout` defaults when set, applying to **both**
`Install.Timeout` and `Upgrade.Timeout`. kubernetes-rd and tenant-rd
carry it today (Kamaji bootstrap; seaweedfs-db CNPG bootstrap).
A new `release.cozystack.io/helm-upgrade-timeout` annotation overrides
**only** `Upgrade.Timeout`, winning over the value the install
annotation would otherwise apply — so a kind can carry an asymmetric
install/upgrade budget. No `ApplicationDefinition` sets it today, so
behavior is unchanged for every existing kind.
`Install.Timeout` and `Upgrade.Timeout` are now always populated (from a
per-app override or the global default), matching the operator's
`buildHelmReleaseSpec`. The previous "stay nil when zero" branch was
removed — it was a test shim that kept empty-`ReleaseConfig{}` fixtures
green at the cost of api/operator drift.
### Shared validator
`parsePositiveDuration` is lifted from `cmd/cozystack-operator/main.go`
into `pkg/config` (exported as `ParsePositiveDuration`). Both
`cozystack-operator` and `cozystack-api` mains now share one validator
with identical reject rules (zero / negative / malformed / empty). The
table-driven test moves with it. The annotation parser is shared the
same way: `ParseHelmTimeoutAnnotation` validates both the install- and
upgrade-timeout annotation values against Flux's unit rules.
## Production safety
- Static flag defaults match #2509 exactly. **With stock defaults, every
api-generated HR changes shape**: it gains
`Strategy.Name=RetryOnFailure` + `RetryInterval`, an explicit
`MaxHistory`, and an explicit `Install/Upgrade.Timeout`. This is the
intended parity with the operator path, not a no-op.
- **Timeout-default change**: because `Install/Upgrade.Timeout` is now
always set, the effective install/upgrade timeout for api-generated HRs
grows from the implicit flux default (5m, field was previously nil) to
the explicit `--helmrelease-{install,upgrade}-timeout` default of `10m`.
This is desired — it matches the operator path.
- Same intentional retry-cadence change as #2509: failed-release retry
cadence on api-generated HRs drops from ~5m (previously coupled to
`Spec.Interval`) to 30s (`RetryInterval` default). Healthy-release
reconcile cadence is unchanged at 5m.
- Per-Application `HelmInstallTimeout` override semantics intact —
kubernetes-rd's long Kamaji bootstrap path continues to work;
tenant-rd's seaweedfs-db bootstrap budget is unchanged.
## Verification
- `go build ./...` clean
- `go vet ./...` clean
- `go test ./pkg/config/... ./pkg/cmd/server/...
./pkg/registry/apps/application/... ./cmd/cozystack-operator/...
./internal/operator/...` — all pass
- New / updated unit tests mirror the operator's spec tests:
- `TestConvertApplicationToHelmRelease_BuildsSpecFromConfig` — full HR
shape from config (mirrors `TestBuildHelmReleaseSpec`)
- `TestConvertApplicationToHelmRelease_ZeroMaxHistory` — `MaxHistory=0`
survives as pointer-to-0 (mirrors
`TestBuildHelmReleaseSpecZeroMaxHistory`)
- `TestConvertApplicationToHelmRelease_PerAppTimeoutOverridesGlobal` —
install-annotation override wins over globals (symmetric + asymmetric)
- `TestConvertApplicationToHelmRelease_UpgradeTimeoutAnnotation` —
upgrade-only override, asymmetric precedence over the install
annotation, and the install-sets-both regression guard
- `TestConvertApplicationToHelmRelease_AppliesReleaseConfigTimeout` —
`Remediation.Retries == -1` assertions replaced with `Strategy.Name ==
RetryOnFailure` + `Remediation == nil`
- `TestParsePositiveDuration` / `TestParseHelmTimeoutAnnotation` — flag
and annotation validation, including which error branch fires
- Pre-existing test failures in `pkg/apiserver` (randfill panic) and
`pkg/lineage` reproduce on `main` without these changes — not introduced
here.
## Test plan
- [ ] CI E2E install passes with default values (no flags set on
cozystack-api Deployment)
- [ ] HR objects generated from Application resources in a healthy
cluster render with `Strategy.Name=RetryOnFailure` and
`RetryInterval=30s`
- [ ] kubernetes-rd Application's HR still gets
`Install/Upgrade.Timeout=15m` (or whatever its annotation says) —
overrides the global default
## Followups (deliberately not bundled)
- **Helm chart exposure** — equivalent of #2509's
`packages/core/installer/values.yaml` knobs, but for the api server.
Touches a different chart (`packages/core/cozystack-api` or wherever the
api Deployment lives). Separate PR keeps the diff focused on the Go
side.
- **Per-Application override of the remaining globals** — interval /
retry-interval / max-history have no annotation hook yet; adding them
would require new annotations and is out of scope until a real need
surfaces.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added server-wide HelmRelease timing flags and defaults (interval,
retry interval, install/upgrade timeouts, max history), with
per-application timeout overrides.
* HelmRelease generation now consistently applies a retry-on-failure
strategy with explicit install/upgrade timeouts.
* **Bug Fixes**
* Stricter validation for duration-related flags to reject malformed,
zero, or negative values.
* **Tests**
* Expanded unit tests for flag parsing/validation, HelmRelease spec
generation (including max-history=0 handling), timeout override
precedence, and invalid Git ref handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The "Set up Buildx" step in build-main.yaml ran without DOCKER_CONFIG, so
docker/setup-buildx-action created the docker-container builder under the
default ~/.docker. The build step runs with DOCKER_CONFIG=$RUNNER_TEMP/.docker
(seeded before the builder existed and holding the OCIR auth), so it could not
find the builder and failed immediately with `no builder "builder-<uuid>"
found`. The post-job `docker buildx rm` still succeeded because the action's
cleanup runs under the default config where the builder actually lived.
Set DOCKER_CONFIG on the Set up Buildx step so the builder instance is created
in the same config the build step reads.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The shardCount description and the shard.resources value overflowed their columns, pushing the internal separators out of line. Reflow every column to its widest cell so the raw table renders cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
BuildShardDeployment clones the flux-aio pod spec, which carries a required podAntiAffinity on app.kubernetes.io/name=flux. Inherited onto a shard pod it forbids scheduling on the flux-aio node, so on a single schedulable node every shard stays Pending and the tenant plane stalls (no ready shard -> assignments deferred -> HelmReleases stuck on key=tenants).
Drop the podAntiAffinity during sanitisation while keeping any benign nodeAffinity (e.g. os=linux). The fixture now carries flux-aio's affinity and TestBuildShardDeployment asserts the strip.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
release.cozystack.io/helm-install-timeout sets both Install.Timeout and
Upgrade.Timeout to one value. Add release.cozystack.io/helm-upgrade-timeout
so an ApplicationDefinition can override only Upgrade.Timeout — letting a
kind keep a short install budget but a longer upgrade budget (or vice
versa). When set it wins over the value the install annotation would
otherwise apply to the upgrade side.
- config: add HelmUpgradeTimeoutAnnotation + ReleaseConfig.HelmUpgradeTimeout;
generalize ParseHelmInstallTimeoutAnnotation to ParseHelmTimeoutAnnotation
since it now parses both annotations (identical Flux unit validation).
- server: parse the new annotation at startup with the same loud-fail
behavior; update --helmrelease-upgrade-timeout help.
- rest: apply HelmUpgradeTimeout after the install-annotation copy so the
upgrade-specific value wins; restore the accurate carrier comment.
- tests: cover the upgrade-only override, the asymmetric precedence, and
the install-sets-both regression guard.
Behavior is unchanged for every existing ApplicationDefinition (none set
the new annotation today).
Address review feedback from coderabbitai on rest.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The ParsePositiveDuration table exercised all error inputs but did not
distinguish which branch fired. Add an errMatch field so the table
pins "must be > 0" for zero/negative and "invalid duration" for
malformed/empty inputs, mirroring TestParseHelmInstallTimeoutAnnotation
and keeping the two parsers' diagnostics from drifting.
Address review feedback from coderabbitai on config_test.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
HelmReleaseMaxHistory carried omitempty, so a value of 0 (which Helm
treats as "unlimited") would be dropped on marshal and become
indistinguishable from an unset field. The runtime path (start.go
always populates it from --helmrelease-max-history) is safe today, but
drop omitempty so the explicit "unlimited" intent survives if
ReleaseConfig is ever serialized.
Address review feedback from coderabbitai on config.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
gofmt -w on the two touched test files: the case-table struct fields in
rest_helmrelease_spec_test.go and main_test.go were misaligned, and
main_test.go carried a trailing blank line. CI runs no gofmt gate
(pre-commit only runs make generate), so this would otherwise land
unformatted.
Address review feedback from lexfrei.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil