The e2e runner hack/cozytest.sh converts only @test blocks into functions and runs those; it never invokes bats setup_file/teardown_file. So the healer's setup_file deploy silently never executed (confirmed: the Job was absent from the cozyreport while the same manifest deploys and runs fine on a real Talos cozystack cluster). Move the deploy into a real, first @test so cozytest.sh runs it (with visible set -x trace) before the install churn, and assert the Job landed without ever failing the suite (band-aid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The HR readiness gate in e2e-install-cozystack.bats was toothless in two
layers: the backgrounded kubectl-wait fan-out discarded child exit codes
(POSIX `wait` without args returns 0), and the final check echoed
"Some HelmReleases failed to reconcile" without exiting non-zero — a
regression from 1f240387f, which replaced the unavailable bats `fail`
helper with a bare echo. A permanently-failing platform HelmRelease
(e.g. the backup-controller CRD rejection, #2822) shipped through green
CI for weeks.
Replace the fan-out with a single `kubectl wait hr --all -A` for trace
visibility, gate on an outcome-based re-list (which also covers HRs
created after the snapshot, with a 120s retry to absorb momentary
drift-reconcile flaps), and dump the full Ready condition message per
non-ready HR so the real error is visible in the test output.
Refs #2822
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
external host at container startup. The e2e sandbox uses
host=example.org which does not resolve, so the auth-proxy crashloops:
Get "https://keycloak.example.org/realms/cozy/.well-known/openid-configuration":
dial tcp: lookup keycloak.example.org: no such host
Under Flux v2.8 kstatus the gatekeeper Deployment then flips to
'Failed' (rather than the pre-v2.8 'InProgress') and stalls the
cozy-dashboard/dashboard HelmRelease, which in turn blocks
cozy-fluxcd/flux-plunger via dependsOn. Previous runs masked this with
an unrelated ImagePullBackOff on cozy-dashboard-console; with the
console image fixed the OIDC discovery failure now surfaces.
packages/core/platform/values.yaml:194-198 already declares
authentication.oidc.keycloakInternalUrl. When set, the dashboard
chart (packages/system/dashboard/templates/gatekeeper.yaml:59-66)
renders --skip-oidc-discovery plus explicit redeem/jwks/validate/
backend-logout URLs pointing at the in-cluster keycloak Service.
Browser-facing redirect-url and login-url stay on the external host
(they are not exercised by the e2e flow).
The patch sets the value alongside oidc.enabled in the same
kubectl-patch so the configmap and gatekeeper rollout happen in one
reconcile pass.
Observed on CI run 26142986969 (PR #2619).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5915a579ee)
## What this PR does
Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical
platform
objects labeled `platform.cozystack.io/no-delete=true`. Native,
in-process CEL
evaluation — no webhook DaemonSet, no Service, no TLS, no image to
maintain.
Requires Kubernetes 1.30+.
This is an alternative to the closed#2402 (admission webhook approach).
### Protected objects (labeled in this PR)
| Resource | Kind | Source |
|---|---|---|
| `cozy-system` | Namespace |
`packages/core/installer/templates/cozy-system-{namespace,labels}.yaml`
|
| `tenant-root` | Namespace |
`packages/system/cozystack-basics/templates/tenant-root.yaml` |
| `tenant-root` | HelmRelease (ns: tenant-root) | same |
| `cozystack-version` | ConfigMap (ns: cozy-system) |
`packages/core/platform/templates/cozystack-version.yaml` |
| `cozystack-packages` | OCIRepository (ns: cozy-system) |
`packages/core/platform/templates/repository.yaml` |
| `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` |
ClusterIssuer |
`packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` |
| `linstorcluster` | LinstorCluster |
`packages/system/linstor/templates/cluster.yaml` |
| `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label
stamped in `internal/crdinstall/install.go` |
### Bypass
```bash
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
```
### Why VAP instead of a webhook
Same outcome as #2402 but with zero runtime infrastructure: no
DaemonSet, Service,
TLS certificate, or image. The kube-apiserver evaluates the CEL
`expression: "false"` in-process and denies because `validationActions:
[Deny]`.
`objectSelector` on the binding scopes evaluation to labeled objects
only — every
other DELETE in the cluster is unaffected.
### Release note
```release-note
feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy)
blocking DELETE on cozystack platform objects labeled
`platform.cozystack.io/no-delete=true`. Bypass by removing the label first.
Requires Kubernetes 1.30+.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Deletion-protection guardrail: objects labeled
platform.cozystack.io/no-delete=true are denied DELETE requests; removal
of the label allows deletion.
* **Chores**
* Platform migration to v43 backfills the no-delete label onto system
resources and namespaces.
* **Tools**
* Added an "unprotect" teardown utility to remove the no-delete label
for uninstall/disaster recovery.
* **Documentation**
* Helm NOTES now warns when the guardrail is inactive due to missing
cluster API.
* **Tests**
* New tests and e2e checks validate deletion-protection behavior.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Without pipefail (dash-CI), `yq | hack/e2e-prepull-images.sh` would let a
yq failure produce empty input; prepull-images.sh exits 0 on no-input
and the test passes with pre-pull silently skipped. Stage the yq output
to a tmp file and feed it via stdin so set -e trips on a yq failure.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
unprotect.sh: add `gitrepository` to the namespaced-kind list so the
platform stays unprotectable when an operator configures
`sourceRef.kind: GitRepository`. Currently only `ocirepository` is
swept, while repository.yaml templates `kind: {{ $sourceRef.kind }}`
unrestricted.
run-migrations.sh: replace the silent skip on a missing migration file
with a hard error. All migrations 1..targetVersion exist in the shipped
image — a gap can only mean a packaging mistake (digest advanced
without the file landing), which previously stalled the cluster at a
stale version while reporting success.
migrations/42: drop the misleading "no 3-way-merge subtlety" claim from
the header. The apply records the no-delete label under the default
field manager's last-applied annotation, so a future label-less apply
by the same field manager would strip it. Document the constraint
instead of denying it.
hack/e2e-install-cozystack.bats: restore `helm.sh/resource-policy: keep`
on the reconstructed cozystack-version ConfigMap in the bypass-path
teardown so cluster state matches the chart template shape
(packages/core/platform/templates/cozystack-version.yaml) on exit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml: the VAP contract had no integration coverage.
A unit test that greps the rendered manifest is the wrong shape — the
contract is "the api-server denies a labeled DELETE with the documented
message" and that needs a real apiserver in the loop.
Add a post-install bats test to hack/e2e-install-cozystack.bats that runs
once cozystack is installed on the e2e cluster:
1. Skips on clusters that pre-date Kubernetes 1.30 (no VAP API).
2. Asserts the cozystack-version ConfigMap actually carries
platform.cozystack.io/no-delete=true — precondition, so the deny
assertion below would not misreport on a regressed binding.
3. Asserts `kubectl delete configmap cozystack-version -n cozy-system`
exits non-zero AND its error message contains the documented
"Deletion blocked: ... platform.cozystack.io/no-delete=true"
substring AND the `--namespace` bypass hint.
4. Confirms the ConfigMap survives the rejected delete.
5. Removes the label, deletes (must succeed), recreates the ConfigMap
with the original data.version and the label so the cluster ends
the test in the same state it started.
This single test catches every regression the PR is meant to prevent:
capability gate inverted, binding objectSelector mistyped,
validationActions flipped Deny→Warn, expression flipped false→true,
label-key drift between the binding and the manifests, and the bypass
docs missing --namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Cluster-member workloads (OVN raft, LINSTOR, cert-manager) fail when
replicas start at different times due to per-node image-pull stagger.
This test pre-pulls the images that those charts reference onto every
node before the platform HelmReleases install, so all replicas can
start with images already cached.
Mechanics:
- Source images directly from the rendered charts (kubeovn, linstor,
cert-manager) via `helm template` + a `yq` filter that walks every
PodSpec-shaped object and emits each container's image. Scopes the
output to images the kubelet actually pulls — skips configmap fields
and CRD examples that happen to contain an `image:` key. Each render
is staged through a tmp file so a helm-template failure trips `set -e`
cleanly without needing `pipefail` (cozytest.sh runs @test bodies
under /bin/sh, which is dash on Ubuntu CI), and without capturing
rendered YAML into a shell variable that `set -x` would expand and
balloon the CI trace.
- The pre-pull DaemonSet is the hard part: distroless images
(cert-manager and friends) ship no shell and no /bin/sleep, so the
obvious `command: ["sleep", "infinity"]` fails with exec ENOENT.
Stage a statically-linked busybox:musl into a shared emptyDir from
an initContainer; every prepull container then execs /shared/sleep
regardless of what the image itself ships. The glibc busybox tag is
unusable here — it needs /lib64/ld-linux-x86-64.so.2 which distroless
lacks. Loop is written in POSIX shell (no bash arrays) for the same
/bin/sh-is-dash reason.
Verified locally on kind v1.33.1 single-node: every chart-referenced
image is held Ready under the DaemonSet before the platform HelmRelease
applies.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The prior commit added `set -o pipefail` to catch yq failures in the
prepull pipeline. CI failed at runtime: hack/cozytest.sh has a
`#!/bin/sh` shebang, /bin/sh is dash on Ubuntu CI, and dash rejects
`set -o pipefail` ("Illegal option").
Drop the pipe entirely. Each step is now its own command with stdin/
stdout redirection — set -e catches failures at the helm-template, yq,
or prepull stage without needing pipefail and without `var=$(...)`
captures that would balloon the `set -x` trace with rendered YAML.
Verified under dash in Ubuntu 24.04 by running cozytest.sh against the
test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous form staged each helm-template render to a tmp file so a
render failure would trip `set -e` outside the pipe. That avoided the
`var=$(...)` trace-bloat trap, but yq still ran inside a pipe with
pipefail off, so a yq failure (bad expression, schema drift) would be
masked by the prepull script's exit code and the test would pass while
silently pre-pulling nothing.
Replace the tmp-file dance with a brace group + `set -o pipefail`:
helm-template output flows through the pipe, never expanded into the
`set -x` trace (verified under `hack/cozytest.sh`: 5 command lines, no
YAML), and pipefail now surfaces a failure at any stage. Drops the
mktemp/rm scaffolding and the `cat` subprocess.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The pre-pull test captured each rendered chart into a shell variable
(`kubeovn_manifests=$(helm template ...)`) so a render failure would
trip `set -e` (bats has no `pipefail`, so piping helm into yq would mask
a helm failure with yq's exit code).
cozytest.sh runs every bats test under `set -x`, and bash expands
captured command output into the trace as `+ var=<entire stdout>`.
Two full helm renders (kubeovn + linstor) plus a `printf` re-expansion
were ballooning the install step to ~40k log lines per CI run.
Stage each render to a tmp file instead. The same exit-code semantics
apply (a failed `helm template ... > file` still trips `set -e`), but
the trace now only shows the commands themselves, not the rendered YAML.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
### Parallel post-install prep
LINSTOR / StorageClass / MetalLB setup, previously sequential `@test`
cases after the 15 m platform HR wait, now run as a background prep that
completes during the platform reconcile. None of them gates platform HR
reconcile, so they can move out of the critical path.
New helper `hack/e2e-post-install-prep.sh` polls for its prerequisites
and is awaited at the end of the `Install Cozystack` test so failures
still surface deterministically.
### T2-race fix
Wait on the LINSTOR HR being `Ready` before polling for
`deploy/linstor-controller`. The old 5-min hard timeout fired during
cold installs where the LINSTOR HR's dependency chain (cert-manager →
piraeus-operator-crds → piraeus-operator → linstor) takes longer than 5
min to resolve.
### E2E uses helmReleaseInterval=30s
E2E install passes `--set cozystackOperator.helmReleaseInterval=30s` to
eliminate the 8–10 min Flux dead zone where dependency-blocked HRs are
requeued only every 5 min. Operator-side flag and chart value landed in
#2509; production defaults are unchanged.
Surfaced from #2500.
### Release note
```
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
- Improved end-to-end test initialization to run concurrent post-install
preparation, wait for cluster release reconciliation, verify component
readiness, and validate automated storage configuration.
* **Chores**
- Added background post-install setup and logging to increase test
reliability and speed up cluster initialization and teardown
diagnostics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Replaces fragile `sleep N` / `timeout N until …; do sleep …; done`
patterns in `hack/*.bats` and `hack/*.sh` with `kubectl wait
--for=condition=…` plus a small existence backstop:
```sh
until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
kubectl wait <resource> --for=condition=Ready --timeout=…
```
### Why the existence backstop matters
`kubectl wait` against a not-yet-created resource sits silently for the
full timeout and then fails with `error: timed out waiting for the
condition` — useless for diagnosis. With the backstop, the test fails
fast with a clear "resource never appeared" error.
Existence backstop added at 14 sites in this sweep across:
- `redis`, `kafka`, `openbao`, `harbor`, `mongodb`, `qdrant`,
`clickhouse`, `mariadb`, `postgres`, `bucket`, `external-dns`,
`foundationdb` bats
- `hack/e2e-install-cozystack.bats`
- `hack/e2e-apps/run-kubernetes.sh`
- `hack/e2e-test-openapi.bats` (port-readiness wait replacing fixed
`sleep 5` after `kubectl proxy`)
The vm-disk hunk in `hack/e2e-apps/vminstance.bats` is included here;
the vm-instance hunks land separately via the `daniil/split-vminstance`
PR.
### Genuine sleeps annotated
The 3 remaining genuine sleeps (LINSTOR node membership propagation,
MetalLB end-to-end HTTP path, HR-count emission heuristic) are annotated
with `# TODO` + rationale so a future reader knows they are intentional.
Same green-path semantics; failures now surface in seconds instead of
after a 5-minute timeout. Surfaced from #2500.
### Release note
```
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved e2e stability by replacing fixed sleeps with timeout-based
polling across many integration tests, waiting for operator-created
resources to exist before readiness checks to reduce race conditions and
flakes.
* **Documentation**
* Added clarifying notes for LoadBalancer retry behavior and improved
local proxy readiness checks in e2e scripts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Two complementary changes that depend on the operator's
`--helmrelease-interval` flag landing via #2509:
1. Installer chart exposes `cozystackOperator.helmReleaseInterval`
(default empty -> flag not rendered -> operator default 5m
applies). E2E install passes `30s`. Eliminates the 8-10 min Flux
dead zone where dependency-blocked HRs (waiting on cert-manager
webhooks, CRDs) are requeued only every 5 min.
2. LINSTOR / StorageClass / MetalLB setup, previously sequential
`@test` cases after the 15 m platform HR wait, now run as a
background prep that completes during the platform reconcile.
None of them gates platform HR reconcile, so they can move out
of the critical path. Helper script `hack/e2e-post-install-prep.sh`
polls for its prerequisites and is awaited at the end of the
`Install Cozystack` test so failures still surface deterministically.
Includes the T2-race fix: wait on the LINSTOR HR being `Ready`
before polling for `deploy/linstor-controller`. The old 5-min hard
timeout fired during cold installs where the LINSTOR HR's dependency
chain takes longer than 5 min to resolve.
Production defaults are unchanged. The 30 s interval is opt-in per
install command.
Depends on:
- #2509 (operator-side `--helmrelease-interval` flag).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
## Why
The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:
- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.
This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.
## What
### New operator flags
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.
### Chart exposure
`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.
## Production safety
- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)
## Verification
- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`
## Followups (deliberately not bundled)
- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.
## Test plan
- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.
* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.
* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Why
helm v3 has a chicken-and-egg with charts that ship their own Namespace
resource:
- **With `--create-namespace`** on the install: helm pre-creates the
namespace via plain kubectl-create (no helm meta annotations); the
chart's own Namespace apply then fails with `already exists`.
- **Without `--create-namespace`**: helm fails immediately because it
can't write its release-secret to a non-existent namespace.
The cozy-installer chart shipped a `Namespace cozy-system` resource
since the chart's introduction (December 2025). Every cold install since
has hit this conflict. It was masked by the 3× retry on `Install
Cozystack` in `.github/workflows/pull-requests.yaml`: attempt 1 fails
with the conflict, attempt 2 sees the existing failed release and takes
the upgrade code path which patch-merges instead of strict-create.
Surfaced when retries on the install step were dropped.
## What
- Remove `Namespace cozy-system` from
`packages/core/installer/templates/cozystack-operator.yaml`.
- Add `packages/core/installer/templates/cozy-system-labels.yaml`: a
pre-install/pre-upgrade hook (ServiceAccount + ClusterRole +
ClusterRoleBinding + Job) that patches `cozystack.io/system=true` and
`pod-security.kubernetes.io/enforce=privileged` onto the namespace.
- Hook RBAC is scoped to `verbs: [get, patch]` on `resourceNames:
[cozy-system]` only.
- `hook-delete-policy: before-hook-creation,hook-succeeded` so the hook
surface only exists during install/upgrade.
Install commands now use `helm upgrade --install --namespace cozy-system
--create-namespace …` — standard pattern, matches kube-prometheus-stack
/ argo-cd / cert-manager.
## Verification
End-to-end on a kind cluster:
- Cold install: 3.3s clean, no retries needed
- Upgrade (re-run): 3.0s, idempotent, namespace stays labeled
- Helm uninstall + re-install: clean
## Test plan
- [ ] CI on this PR passes E2E
- [ ] Existing clusters: namespace already labeled correctly, hook
re-asserts on next upgrade — no migration needed
```release-note
fix(installer): cold installs no longer fail on the chart's Namespace resource colliding with `helm --create-namespace`; cozy-system is now bootstrapped via a pre-install hook.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an automated pre-install/pre-upgrade step that applies system
identity and pod-security labels to the system namespace during
deployment, run in a hardened one-shot task before normal workloads
start.
* **Chores**
* Separated namespace initialization from the operator install flow so
the operator no longer renders or retains the system namespace directly,
improving installation sequencing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- kafka.bats: use $name in the kafkas existence backstop, matching the
variable used in the kubectl wait below it (consistency with the rest
of this PR's backstops).
- e2e-install-cozystack.bats: the Tenant→HelmRelease materialisation is
asynchronous (cozystack-controller observes the Tenant CR and creates
the HR), so guard hr/tenant-test with the same existence backstop
pattern used elsewhere.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
With Strategy=RetryOnFailure + RetryInterval=30s plus the bats
--set cozystackOperator.helmReleaseInterval=30s override, the existing
2m wait converges without the manual flux reconcile --force band-aid.
Removing the workaround turns CI itself into the test of the fix.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used
by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks.
- Skip hostNetwork=true and not-ready/CNI tolerations on the hosted
variant: managed clusters provide CNI before install, and kube-system
may carry strict PSA there that would reject hostNetwork pods.
- Add packages/core/installer/templates/cozy-system-namespace.yaml gated
on .Values.bareNamespace (default false). The top-level Makefile's
manifests target now passes --set bareNamespace=true and renders
Namespace + operator into _out/assets/cozystack-operator-*.yaml so
the kubectl-apply install path stays self-contained. Helm
install/upgrade callers keep the default and bootstrap via
--create-namespace + the pre-install labeler hook.
- Add a bats assertion in hack/e2e-install-cozystack.bats checking
cozy-system carries pod-security.kubernetes.io/enforce=privileged and
cozystack.io/system=true after helm install.
- Inline comment near hostNetwork=true documenting why dnsPolicy left
at Default (cluster DNS not up at install time;
ClusterFirstWithHostNet would not help).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server)
fail when replicas start at materially different times due to
image-pull stagger across nodes. Add a DaemonSet-based pre-pull step
that runs before `helm install`, ensuring every node has the images
cached so replicas start within milliseconds of each other.
The script accepts image refs on stdin and creates one container per
image so pulls run in parallel — total time = max(any single image)
rather than sum. The bats test sources image refs directly from the
rendered platform charts via `yq`, walking only PodSpec-shaped
objects, so version bumps stay in sync without a hardcoded list.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Sweep across `hack/*.bats` and `hack/*.sh` replacing fragile
`sleep N` / `timeout N until …; do sleep …; done` patterns with
`kubectl wait --for=condition=…` plus a small existence backstop:
until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
kubectl wait <resource> --for=condition=Ready --timeout=…
Existence backstop added to 33 `kubectl wait` sites across 12 bats
files (redis, kafka, openbao, harbor, mongodb, qdrant, vminstance,
clickhouse, mariadb, postgres, bucket, install). Without it,
`kubectl wait` against a not-yet-created resource sits silently for
the full timeout; with it, the test fails fast with a clear
"resource never appeared" error.
OpenAPI tests' fixed `sleep 5` after starting `kubectl proxy` is now
a port-readiness wait. The 3 remaining genuine sleeps (LINSTOR node
membership propagation, MetalLB end-to-end HTTP path, HR-count
emission heuristic) are annotated with TODO + rationale so a future
reader knows they are intentional.
Same green-path semantics; failures now surface in seconds instead
of after a 5-minute timeout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).
E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.
Production defaults are unchanged. The override is opt-in per install.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Four follow-ups from review round 4:
1. BLOCKER: the Update(forceAllowCreate=true) path delegates to Create() when the object does not yet exist (rest.go:452) — the typical kubectl apply upsert flow. Add TestUpdate_ForceAllowCreate_RejectsTenantDashName using a fake client so a future refactor of that delegation cannot silently bypass the tenant name check that r.validateNameFormat alone cannot catch.
2. BLOCKER: the e2e BATS test used || true inside the command substitution, which swallowed the kubectl exit code. Rework the test to capture exit code and stdout+stderr explicitly, then assert the exit code is non-zero before asserting on the error message. This distinguishes validation-success (kubectl exit 0 — regression) from environmental failures (exit non-zero but wrong message) from the happy path.
3. Extract TenantKind = "Tenant" as a named constant in the validation package with a comment pointing at the upstream ApplicationDefinition source of truth, and switch the kindName check to use it.
4. Add a clarifying comment on TestValidateApplicationName_TenantLengthFallthrough that it pins an architectural layering decision and is not a user-facing requirement, so a future promotion of tenant length into tenant-specific wording is a legitimate change rather than a test regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three follow-ups from review:
1. Add defensive cleanup of the foo-bar tenant before and after the e2e regression test. If a prior run left the object in the cluster (e.g. a transient validation regression), the test no longer inherits that state; and if the test itself trips a regression in the future, the cluster is left clean for subsequent tests.
2. Switch kubectl --validate=false to --validate=ignore. The bool form was deprecated in kubectl 1.25 and only kept as an alias; --validate=ignore is the modern, stable spelling.
3. Drop the brittle Contains("63") assertion in TestValidateApplicationName_TenantLengthFallthrough. It pinned the test to the exact DNS-1035 error text from k8s.io/apimachinery, which could break on unrelated upstream wording changes. The surviving assertions still cover the intent: an error exists and it is not the tenant-specific message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Follow-up from review round 2:
The BATS regression test now pins a namespace precondition, disables kubectl client-side validation with --validate=false so the request is guaranteed to reach the server-side name check, and documents the intent of each assertion. Previously a client-side schema rejection could have produced a false negative on the tenant-specific error grep, and a missing tenant-root namespace could have produced an unrelated failure that was hard to diagnose.
Also pin the edge case where a tenant name consists of valid characters but exceeds the DNS-1035 63-char label limit. The resulting error is intentionally the generic DNS-1035 message because length is not a tenant-specific constraint — the package-level function is not responsible for the stricter Helm-release-prefix length budget that REST.validateNameLength enforces.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two follow-ups from review:
1. The original tenantNameRegex ^[a-z0-9]+$ let leading-digit names like 123foo slip past the tenant check and receive a generic DNS-1035 error, which defeated the goal of surfacing a tenant-specific message. Require a leading lowercase letter so every tenant-invalid name returns the tenant-contract error.
2. Add a BATS regression test in hack/e2e-install-cozystack.bats that exercises the aggregated API end-to-end. Unit tests construct REST{kindName: "Tenant"} by hand, so a future change to ApplicationDefinition kind registration could break real behavior without breaking the unit tests.
Also pin the error-message contract with a new TestValidateApplicationName_TenantErrorMessage that asserts every tenant-invalid input returns a message containing 'tenant names must' (not the generic DNS-1035 text).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
On resource-constrained CI runners, HelmReleases may take close
to 15 minutes to reconcile, leaving very little time for CAPI
deployments to start and become available. Increase timeouts:
- CAPI deployment discovery: 60s -> 180s
- CAPI deployment availability: 1m -> 5m
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
Add external-dns as a standalone self-managed application in
`packages/extra/external-dns/`, allowing tenants to deploy and configure
their own DNS management directly from the dashboard.
## Motivation
Tenants need the ability to manage their own DNS domains with their own
provider. Following the [developers
guide](https://github.com/cozystack/website/pull/413), this is
implemented as an extra package (like `ingress` and `seaweedfs`) using
the HelmRelease-based pattern, rather than embedding it in the tenant
chart.
This enables multi-tenant scenarios where:
- Tenant A uses Cloudflare for `domain-a.com`
- Tenant B uses AWS Route53 for `domain-b.com`
- Each tenant deploys and manages external-dns independently from the
dashboard
## Changes
- **New package**: `packages/extra/external-dns/` — standalone
HelmRelease-based application
- **New PackageSource**:
`packages/core/platform/sources/external-dns-application.yaml` —
references `system/external-dns` and `extra/external-dns` components
- **Cleaned tenant chart**: removed the previously embedded
`externalDns` block from `packages/apps/tenant/`
## Features
- Support for 9 DNS providers: cloudflare, aws, azure, google,
digitalocean, linode, ovh, exoscale, godaddy
- Per-provider credential configuration with full JSON schema validation
- Domain filtering via `domainFilters`
- Configurable sync policy (`sync` or `upsert-only`)
- Namespaced operation (`namespaced: true`) for tenant isolation
- Unique `txtOwnerId` per namespace to prevent DNS record conflicts
- Resource sizing via presets or explicit CPU/memory
## Usage Example
Deploy from the dashboard, or via values:
```yaml
# Cloudflare
provider: cloudflare
domainFilters:
- example.com
cloudflare:
apiToken: "your-cloudflare-api-token"
```
```yaml
# AWS Route53
provider: aws
domainFilters:
- example.org
aws:
accessKeyId: "AKIAXXXXXXXX"
secretAccessKey: "your-secret-key"
region: "us-east-1"
```
## Test plan
- [ ] `helm template external-dns packages/extra/external-dns/ --set
provider=cloudflare --set cloudflare.apiToken=test` renders correctly
- [ ] `helm template external-dns packages/extra/external-dns/` fails
(provider required)
- [ ] `helm template wrong-name packages/extra/external-dns/ --set
provider=cloudflare` fails (release name check)
- [ ] Deploy external-dns from tenant dashboard
- [ ] Verify HelmRelease is created in tenant namespace with namespaced
RBAC
- [ ] Create an Ingress and verify DNS record is created
- [ ] Verify no conflict with global external-dns instance
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an External DNS package for automatic DNS record management.
* Support for 9 DNS providers: Cloudflare, AWS, Azure, Google,
DigitalOcean, Linode, OVH, Exoscale, GoDaddy.
* Helm-based deployment with namespaced/system variants and release
configuration options.
* Configurable synchronization policies, domain filtering, provider
credentials, extra args, and resource presets.
* **Documentation**
* New README and schema-driven values documentation for installation and
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The victoria-metrics-operator v0.68.1 renamed VMCluster status field
from .status.clusterStatus to .status.updateStatus.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
After migrating VictoriaLogs from VLogs to VLCluster, the e2e test
still waited for the old vlogs/generic resource which no longer exists.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
kubectl wait fails immediately with NotFound if the CRD does not exist
yet. The operator creates CRDs asynchronously on startup, so wrap the
wait in a retry loop that tolerates the initial absence.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Explicitly check error from parsePlatformSourceURL instead of relying on
the implicit guarantee that installPlatformSourceResource already checked
it. This prevents latent bugs if startup order is ever restructured.
Add wait for platform PackageSource existence in E2E test before creating
Package resource, preventing flaky failures when operator startup is slow.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add explicit CRD wait (kubectl wait crd --for=condition=Established) in
E2E test before creating Package resources, preventing race condition
between operator CRD installation and resource creation.
Add unit tests for installPlatformPackageSource covering create, update,
and GitRepository sourceRef kind scenarios.
Document that hardcoded variant list is an intentional design choice
matching the previous Helm template behavior.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the Helm hook approach with programmatic PackageSource creation
in the operator startup sequence. Helm hooks are unsuitable for persistent
resources like PackageSource because before-hook-creation policy causes
cascade deletion of owned ArtifactGenerators during upgrades.
The operator now creates the platform PackageSource after installing CRDs
and the Flux source resource, using the same create-or-update pattern as
installPlatformSourceResource(). The sourceRef.kind is derived from the
platform source URL (OCIRepository for oci://, GitRepository for git).
Also fix stale comment in e2e test referencing deleted crds/ directory.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm installs crds/ contents before processing templates, resolving the
chicken-and-egg problem where PackageSource CR validation fails because
its CRD hasn't been registered yet.
- Move definitions/ to crds/ in the installer chart
- Remove templates/crds.yaml (Helm auto-installs from crds/)
- Update codegen script to write CRDs to crds/
- Replace helm template with cat for static CRD manifest generation
- Remove pre-apply CRD workaround from e2e test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm cannot validate PackageSource CR during install because the CRD
is part of the same chart. Pre-apply CRDs via helm template + kubectl
apply --server-side before running helm upgrade --install.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add -talos suffix to the default variant output file for consistency
with -generic and -hosted variants. Update all references in CI
workflows, e2e tests, upload scripts, and testing Makefile.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
- Add pattern rule for building cozypkg binaries per platform
(assets-cozypkg-<os>-<arch>) with checksums generation
- Add Version variable to cozypkg CLI injected via ldflags
- Split manifests into separate cozystack-crds.yaml and
cozystack-operator.yaml files
- Update CI workflow to handle CRDs and operator as separate artifacts
- Update e2e tests and upload script for new manifest structure
- Suppress git describe stderr when no tags exist
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
