## What this PR does
Addresses #3513.
When tenant Kubernetes workers miss the 18-minute Ready deadline, the
E2E runner now mints a one-hour `os:reader` Talos client certificate
from the tenant's existing issuer and runs authenticated `talosctl` from
a hardened same-namespace diagnostics Pod. It captures bounded `dmesg`
and kubelet output for every worker VMI in the cozyreport snapshot,
records missing IPs and command exit codes, continues across unreachable
workers, and preserves the original test failure.
The diagnostics Pod has no service-account token or Secret volume;
credentials are streamed into an `emptyDir`, and the Certificate,
Secret, and Pod are labeled for cleanup. Workers that have not reached
Talos `apid` can only yield a bounded connection error, while workers
that reached `apid` provide guest logs. This adds the evidence needed to
distinguish the remaining guest boot failure but does not claim to
resolve its underlying cause. The readiness budget and success path are
unchanged.
Because this changes a shared E2E helper, test-impact analysis selects
the full E2E suite for this PR.
### Screenshots
Not applicable; no UI changes.
### Testing
- `hack/cozytest.sh hack/run-kubernetes-talos-diagnostics_test.bats` (10
tests)
- `hack/cozytest.sh hack/run-kubernetes-drain_test.bats`
- `hack/cozytest.sh hack/cozyreport-talos.bats`
- `bash -n hack/e2e-chainsaw/_lib/run-kubernetes.sh`
- `git diff --check origin/main...HEAD`
The full `make bats-unit-tests` run reached the unrelated
`hack/migration-seaweedfs-db-adopt.bats` suite and could not continue
because Docker is unavailable in the development environment. A live
failure-path run was not performed because the available development
environment has no Talos tenant fixture; this PR is intended to exercise
that path in Cozystack CI.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(e2e): capture authenticated Talos kernel and kubelet diagnostics when tenant Kubernetes workers miss the Ready deadline
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved troubleshooting when worker nodes fail to join by collecting
per-node operating system and kubelet diagnostics.
* Diagnostics now continue across partial failures and clean up
temporary resources afterward.
* Extended Kubernetes test operation timeouts to allow failure
diagnostics to complete reliably.
* **Tests**
* Added regression coverage for diagnostic collection, timeout handling,
multi-worker scenarios, missing node addresses, and cleanup.
* **Chores**
* Ensured the required diagnostic image is preloaded on nodes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
condition=Available answers whether the Deployment sits within the
slack its rollout strategy allows, not whether the operator came up.
That is the wrong question for an install gate, and it stays wrong
whatever the strategy is: any maxUnavailable equal to replicas drives
the minimum available replica count to 0 and the condition then reports
True with no running pod. Measured against an operator stuck in
ImagePullBackOff, the old check passed in 156 ms, and the install went
on to fail two minutes later on the CRD wait, which pointed the
investigation at the wrong component.
rollout status reads updated and available replicas directly and only
succeeds once the updated pod is available, which holds under any
rollout strategy. Dump the deployment, pods, operator logs and events
on failure so the cause is legible where it is detected.
Drop --wait from the helm step. Helm 3 decides a Deployment is ready
with the same replicas - maxUnavailable arithmetic, so on the current
strategy it is satisfied at zero ready pods and gates nothing; were the
strategy tightened it would instead fail the install on its own 2m
budget before this gate runs, hiding the diagnostics behind a bare Helm
timeout. That arithmetic is specific to Helm 3, since Helm 4 reworked
--wait into a wait strategy with a different default, so treat the
reason as version-bound. The conclusion is not: one gate that reports
what it saw is better than two that can disagree about whose budget
expired first, whichever major is installed. Hooks are waited for with
or without the flag. The 5m budget here covers scheduling and a cold
pull of an image that is not in the prepull set.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
authentication.oidc.keycloakInternalUrl defaulted to an empty string, which
left oauth2-proxy performing OIDC discovery against the external Keycloak
hostname. That request resolves through public DNS and lands on the root
ingress controller, which does not serve yet on a fresh install: the root
tenant's IngressClass only exists once tenant-root is ready, and tenant-root
is still reconciling while the dashboard starts. Discovery has nothing to talk
to, the gatekeeper fails, and the dashboard never comes up.
The bundled Keycloak is the only supported provider — the external hostname is
templated as keycloak.<host> and the realm is fixed — so backend-to-backend
calls have no reason to leave the cluster. Default the value to
http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service
address keycloak-configure already uses. The dashboard and linstor-gui
gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo
and logout requests to the Service, while browser redirects stay on the
external host. Those calls travel as plaintext HTTP between pods, as
keycloak-configure's admin calls to the same address already do.
The value remains overridable for a Keycloak that does not live at the standard
in-cluster address, and setting it back to an empty string restores external
discovery. Existing OIDC clusters that never set it get a gatekeeper rollout
onto the internal path on upgrade.
Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so
the E2E install now drives that swap to completion: it waits for the auth-proxy
container to appear, then gates on the gatekeeper rollout. Readiness alone
proves nothing there — the HelmRelease is still Ready from the token-proxy
install, and the Deployment keeps the old ReplicaSet Available while a new pod
crashloops, so only the rollout catches a regression back to external discovery.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).
Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.
Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
tenant-test -> mirror pod in kube-system), applied at point-of-use once
Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
must get 206 - the exact egress path a real importer faces; tenants use the
mirror only when that end-to-end check passes, else fall back to the public
factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
strict 206 gate, and the --overrides JSON builder.
Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The kubernetes-* e2e tests fail intermittently and run 18-22 min each
because every tenant worker VM imports its Talos OS disk directly from the
public factory.talos.dev: that endpoint has no byte-range support (forcing
CDI into a full copy-to-scratch) and its egress stream-resets / stalls
mid-transfer from the CI runner, so a worker DataVolume import hangs past
the chart's 12-minute node-join deadline and the test fails with zero
tenant nodes joined (observed on run 28924455578, and on the unrelated
branch in run 28929095889 - i.e. not specific to any one PR).
Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml)
during install: an initContainer fetches the image once with an aggressive
curl retry loop, then busybox httpd serves it locally with range support.
Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via
a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the
mirror being Available and falls back to the public factory otherwise, so
the mirror can only help, never make CI worse. Deployed early so the seed
overlaps the Cozystack install; readiness is resolved once and cached
across bats files. Uses the overridable chart value added in the previous
commit.
Refs #3231
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed
test-apps-% target; switch it to the single chainsaw invocation
(empty CHAINSAW_SUITES = full suite) and upload the JUnit report
- pin the chainsaw-report upload-artifact step to the same SHA the
rest of the workflows use (scorecard/zizmor gate)
- repoint hack/run-kubernetes-drain_test.bats at the relocated
hack/e2e-chainsaw/_lib/run-kubernetes.sh
- replicate cozytest.sh's NotReady-pod data-plane capture
(hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so
chainsaw-suite failures keep diagnostic parity with the BATS lanes
- refresh stale bats path comments
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The E2E sandbox tenant (tenant-test) provisions back-to-back tenant
Kubernetes control planes via kubernetes-latest.bats and
kubernetes-previous.bats. Each control plane brings up two worker VMs
whose 20Gi root disks are populated by CDI, and CDI's second-phase
importer allocates a scratch PVC of the same size before the qcow2
unpack. Peak per-tenant footprint is therefore ~82Gi during import
(2 × 20Gi target + 2 × 21Gi scratch), collapsing to 40Gi at rest.
kubernetes-latest teardown deletes the tenant Kubernetes CR and waits
via cozy_wait_tenant_drained for the KubeVirt VMs, VMIs and namespace
PVCs to disappear. The API-level PVC delete returns quickly, but the
LINSTOR CSI detach + DRBD resource release on the underlying nodes
lags past the 300s drain budget. During that window, the deleted
PVCs still count against ResourceQuota's requests.storage because
the storage-provisioner has not finished releasing the volume — so
kubernetes-previous starts its own CDI import with 40Gi of stale
kubernetes-latest quota still charged, and CDI's scratch PVC create
for the second worker (used=80Gi + scratch=21Gi = 101Gi) is
rejected by the tenant-quota admission check at 100Gi. The second
worker's DataVolume never leaves ImportInProgress, the tenant node
never joins, and the 12-minute node-join deadline in
run-kubernetes.sh trips kubernetes-previous.bats.
Raise the sandbox tenant-quota storage ceiling to 200Gi so the
overlap between one test's slow storage cleanup and the next test's
CDI import fits inside the quota. The failing-run diagnostic is
cozyreport artifact from run 28730757213:
cdi-deployment log:
scratch PVC API create errored: … forbidden: exceeded quota:
tenant-quota, requested: requests.storage=22763536384,
used: requests.storage=85899345920,
limited: requests.storage=100Gi
This is a test-infrastructure fix; the 100Gi default in the tenant
chart itself is unchanged. The assertion below the CR update is
lifted from 100Gi to 200Gi in lockstep.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The Configure-Tenant install test completes the tenant-root monitoring
bring-up by waiting for the grafana-db CNPG cluster and the
grafana-deployment Deployment. Both used a 10m budget while the
VictoriaMetrics waits above them use 15m. They complete the same
monitoring bring-up and contend for the same node resources during
install, so under install-time load (concurrent e2e sandboxes on one
runner) either can be slow, and the 10m/15m split within the same
contend-for-resources bring-up risks failing the whole install on a PR
that never touched monitoring.
Raise both Grafana waits to the same uniform 15m budget as the
vm-operator waits and wrap them in an on-timeout diagnostic block that
dumps the resource YAML plus tenant-root pod state, so a genuinely stuck
resource still fails at 15m with legible diagnostics instead of a bare
"timed out" line. The extra budget costs near-zero wall-time on the happy
path and stays inside the E2E job budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
vlclusters/generic and vmcluster wait for updateStatus=operational with
a 10m budget, while their vmalert/vmalertmanager sibling in the same
block already uses 15m. All three are vm-operator-managed monitoring
resources that only become operational once their workloads schedule and
turn Ready, and during platform bring-up they contend for node resources
with the rest of the install. On a calm sandbox each converges in under a
second, but under install-time load (concurrent e2e sandboxes on one
runner) monitoring bring-up is slow, and a 10m/15m split inside a block
whose resources all contend for the same node capacity risks failing the
whole install on a PR that never touched monitoring.
Unify vlclusters and vmcluster on the sibling's 15m budget so the whole
block shares one headroom, and dump resource YAML plus pod state on
timeout across all three waits so a genuine stuck-not-slow regression
stays legible instead of a bare "timed out" line. The extra budget costs
near-zero wall-time on the happy path and stays inside the E2E job
budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The e2e install still flakes on a busy runner where components reach
readiness but overshoot tight wait budgets:
- keycloak HR chain (keycloak-db CNPG -> keycloak StatefulSet ->
keycloak-configure) reaches Ready ~15-18m in: 10m -> 20m.
- tenant aggregate wait (hr/etcd hr/ingress hr/monitoring hr/seaweedfs
hr/tenant-root): the tenant stack only starts creating pods ~9-10m in
and then needs several more minutes: 10m -> 20m.
- per-app tenant readiness waits (ingress, etcd, VictoriaLogs/Metrics
clusters, grafana-db, grafana): 5m -> 10m.
Diagnosed via a live SSH breakpoint: the HRs all do reach Ready, just
outside the old budgets.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
These bats files are all run by hack/cozytest.sh, which executes each
@test under `set -e`. POSIX errexit is suppressed for a pipeline that
begins with `!`, so a bare `! cmd` must-not-happen assertion never fails
the test: when the forbidden condition actually occurs (cmd succeeds) the
negated pipeline returns non-zero but errexit ignores it and the test
passes anyway. These assertions were silently vacuous.
Replace each with `if cmd; then echo "<reason>"; false; fi`, the idiom
already used and documented elsewhere in these files — the if-condition
is exempt from errexit and the `false` in the taken branch fails the test
loudly. The forbidden condition now fails the test as intended:
- bucket: read-only user must not be able to upload (access control)
- gateway: child tenant must not have its own gateway HelmRelease
- kuberture: external-dns public/internal hostname isolation
- e2e-install: tenant-name validation must reject (not warn); the version
ConfigMap must be gone after delete
- build-matrix: talos/installer excluded from the parallel matrix
- capture-dataplane: kube-dns and pending rows are filtered out
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).
Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.
RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The e2e runner hack/cozytest.sh converts only @test blocks into functions and runs those; it never invokes bats setup_file/teardown_file. So the healer's setup_file deploy silently never executed (confirmed: the Job was absent from the cozyreport while the same manifest deploys and runs fine on a real Talos cozystack cluster). Move the deploy into a real, first @test so cozytest.sh runs it (with visible set -x trace) before the install churn, and assert the Job landed without ever failing the suite (band-aid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The HR readiness gate in e2e-install-cozystack.bats was toothless in two
layers: the backgrounded kubectl-wait fan-out discarded child exit codes
(POSIX `wait` without args returns 0), and the final check echoed
"Some HelmReleases failed to reconcile" without exiting non-zero — a
regression from 1f240387f, which replaced the unavailable bats `fail`
helper with a bare echo. A permanently-failing platform HelmRelease
(e.g. the backup-controller CRD rejection, #2822) shipped through green
CI for weeks.
Replace the fan-out with a single `kubectl wait hr --all -A` for trace
visibility, gate on an outcome-based re-list (which also covers HRs
created after the snapshot, with a 120s retry to absorb momentary
drift-reconcile flaps), and dump the full Ready condition message per
non-ready HR so the real error is visible in the test output.
Refs #2822
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
external host at container startup. The e2e sandbox uses
host=example.org which does not resolve, so the auth-proxy crashloops:
Get "https://keycloak.example.org/realms/cozy/.well-known/openid-configuration":
dial tcp: lookup keycloak.example.org: no such host
Under Flux v2.8 kstatus the gatekeeper Deployment then flips to
'Failed' (rather than the pre-v2.8 'InProgress') and stalls the
cozy-dashboard/dashboard HelmRelease, which in turn blocks
cozy-fluxcd/flux-plunger via dependsOn. Previous runs masked this with
an unrelated ImagePullBackOff on cozy-dashboard-console; with the
console image fixed the OIDC discovery failure now surfaces.
packages/core/platform/values.yaml:194-198 already declares
authentication.oidc.keycloakInternalUrl. When set, the dashboard
chart (packages/system/dashboard/templates/gatekeeper.yaml:59-66)
renders --skip-oidc-discovery plus explicit redeem/jwks/validate/
backend-logout URLs pointing at the in-cluster keycloak Service.
Browser-facing redirect-url and login-url stay on the external host
(they are not exercised by the e2e flow).
The patch sets the value alongside oidc.enabled in the same
kubectl-patch so the configmap and gatekeeper rollout happen in one
reconcile pass.
Observed on CI run 26142986969 (PR #2619).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5915a579ee)
## What this PR does
Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical
platform
objects labeled `platform.cozystack.io/no-delete=true`. Native,
in-process CEL
evaluation — no webhook DaemonSet, no Service, no TLS, no image to
maintain.
Requires Kubernetes 1.30+.
This is an alternative to the closed#2402 (admission webhook approach).
### Protected objects (labeled in this PR)
| Resource | Kind | Source |
|---|---|---|
| `cozy-system` | Namespace |
`packages/core/installer/templates/cozy-system-{namespace,labels}.yaml`
|
| `tenant-root` | Namespace |
`packages/system/cozystack-basics/templates/tenant-root.yaml` |
| `tenant-root` | HelmRelease (ns: tenant-root) | same |
| `cozystack-version` | ConfigMap (ns: cozy-system) |
`packages/core/platform/templates/cozystack-version.yaml` |
| `cozystack-packages` | OCIRepository (ns: cozy-system) |
`packages/core/platform/templates/repository.yaml` |
| `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` |
ClusterIssuer |
`packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` |
| `linstorcluster` | LinstorCluster |
`packages/system/linstor/templates/cluster.yaml` |
| `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label
stamped in `internal/crdinstall/install.go` |
### Bypass
```bash
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
```
### Why VAP instead of a webhook
Same outcome as #2402 but with zero runtime infrastructure: no
DaemonSet, Service,
TLS certificate, or image. The kube-apiserver evaluates the CEL
`expression: "false"` in-process and denies because `validationActions:
[Deny]`.
`objectSelector` on the binding scopes evaluation to labeled objects
only — every
other DELETE in the cluster is unaffected.
### Release note
```release-note
feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy)
blocking DELETE on cozystack platform objects labeled
`platform.cozystack.io/no-delete=true`. Bypass by removing the label first.
Requires Kubernetes 1.30+.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Deletion-protection guardrail: objects labeled
platform.cozystack.io/no-delete=true are denied DELETE requests; removal
of the label allows deletion.
* **Chores**
* Platform migration to v43 backfills the no-delete label onto system
resources and namespaces.
* **Tools**
* Added an "unprotect" teardown utility to remove the no-delete label
for uninstall/disaster recovery.
* **Documentation**
* Helm NOTES now warns when the guardrail is inactive due to missing
cluster API.
* **Tests**
* New tests and e2e checks validate deletion-protection behavior.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Without pipefail (dash-CI), `yq | hack/e2e-prepull-images.sh` would let a
yq failure produce empty input; prepull-images.sh exits 0 on no-input
and the test passes with pre-pull silently skipped. Stage the yq output
to a tmp file and feed it via stdin so set -e trips on a yq failure.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
unprotect.sh: add `gitrepository` to the namespaced-kind list so the
platform stays unprotectable when an operator configures
`sourceRef.kind: GitRepository`. Currently only `ocirepository` is
swept, while repository.yaml templates `kind: {{ $sourceRef.kind }}`
unrestricted.
run-migrations.sh: replace the silent skip on a missing migration file
with a hard error. All migrations 1..targetVersion exist in the shipped
image — a gap can only mean a packaging mistake (digest advanced
without the file landing), which previously stalled the cluster at a
stale version while reporting success.
migrations/42: drop the misleading "no 3-way-merge subtlety" claim from
the header. The apply records the no-delete label under the default
field manager's last-applied annotation, so a future label-less apply
by the same field manager would strip it. Document the constraint
instead of denying it.
hack/e2e-install-cozystack.bats: restore `helm.sh/resource-policy: keep`
on the reconstructed cozystack-version ConfigMap in the bypass-path
teardown so cluster state matches the chart template shape
(packages/core/platform/templates/cozystack-version.yaml) on exit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml: the VAP contract had no integration coverage.
A unit test that greps the rendered manifest is the wrong shape — the
contract is "the api-server denies a labeled DELETE with the documented
message" and that needs a real apiserver in the loop.
Add a post-install bats test to hack/e2e-install-cozystack.bats that runs
once cozystack is installed on the e2e cluster:
1. Skips on clusters that pre-date Kubernetes 1.30 (no VAP API).
2. Asserts the cozystack-version ConfigMap actually carries
platform.cozystack.io/no-delete=true — precondition, so the deny
assertion below would not misreport on a regressed binding.
3. Asserts `kubectl delete configmap cozystack-version -n cozy-system`
exits non-zero AND its error message contains the documented
"Deletion blocked: ... platform.cozystack.io/no-delete=true"
substring AND the `--namespace` bypass hint.
4. Confirms the ConfigMap survives the rejected delete.
5. Removes the label, deletes (must succeed), recreates the ConfigMap
with the original data.version and the label so the cluster ends
the test in the same state it started.
This single test catches every regression the PR is meant to prevent:
capability gate inverted, binding objectSelector mistyped,
validationActions flipped Deny→Warn, expression flipped false→true,
label-key drift between the binding and the manifests, and the bypass
docs missing --namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Cluster-member workloads (OVN raft, LINSTOR, cert-manager) fail when
replicas start at different times due to per-node image-pull stagger.
This test pre-pulls the images that those charts reference onto every
node before the platform HelmReleases install, so all replicas can
start with images already cached.
Mechanics:
- Source images directly from the rendered charts (kubeovn, linstor,
cert-manager) via `helm template` + a `yq` filter that walks every
PodSpec-shaped object and emits each container's image. Scopes the
output to images the kubelet actually pulls — skips configmap fields
and CRD examples that happen to contain an `image:` key. Each render
is staged through a tmp file so a helm-template failure trips `set -e`
cleanly without needing `pipefail` (cozytest.sh runs @test bodies
under /bin/sh, which is dash on Ubuntu CI), and without capturing
rendered YAML into a shell variable that `set -x` would expand and
balloon the CI trace.
- The pre-pull DaemonSet is the hard part: distroless images
(cert-manager and friends) ship no shell and no /bin/sleep, so the
obvious `command: ["sleep", "infinity"]` fails with exec ENOENT.
Stage a statically-linked busybox:musl into a shared emptyDir from
an initContainer; every prepull container then execs /shared/sleep
regardless of what the image itself ships. The glibc busybox tag is
unusable here — it needs /lib64/ld-linux-x86-64.so.2 which distroless
lacks. Loop is written in POSIX shell (no bash arrays) for the same
/bin/sh-is-dash reason.
Verified locally on kind v1.33.1 single-node: every chart-referenced
image is held Ready under the DaemonSet before the platform HelmRelease
applies.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The prior commit added `set -o pipefail` to catch yq failures in the
prepull pipeline. CI failed at runtime: hack/cozytest.sh has a
`#!/bin/sh` shebang, /bin/sh is dash on Ubuntu CI, and dash rejects
`set -o pipefail` ("Illegal option").
Drop the pipe entirely. Each step is now its own command with stdin/
stdout redirection — set -e catches failures at the helm-template, yq,
or prepull stage without needing pipefail and without `var=$(...)`
captures that would balloon the `set -x` trace with rendered YAML.
Verified under dash in Ubuntu 24.04 by running cozytest.sh against the
test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The previous form staged each helm-template render to a tmp file so a
render failure would trip `set -e` outside the pipe. That avoided the
`var=$(...)` trace-bloat trap, but yq still ran inside a pipe with
pipefail off, so a yq failure (bad expression, schema drift) would be
masked by the prepull script's exit code and the test would pass while
silently pre-pulling nothing.
Replace the tmp-file dance with a brace group + `set -o pipefail`:
helm-template output flows through the pipe, never expanded into the
`set -x` trace (verified under `hack/cozytest.sh`: 5 command lines, no
YAML), and pipefail now surfaces a failure at any stage. Drops the
mktemp/rm scaffolding and the `cat` subprocess.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The pre-pull test captured each rendered chart into a shell variable
(`kubeovn_manifests=$(helm template ...)`) so a render failure would
trip `set -e` (bats has no `pipefail`, so piping helm into yq would mask
a helm failure with yq's exit code).
cozytest.sh runs every bats test under `set -x`, and bash expands
captured command output into the trace as `+ var=<entire stdout>`.
Two full helm renders (kubeovn + linstor) plus a `printf` re-expansion
were ballooning the install step to ~40k log lines per CI run.
Stage each render to a tmp file instead. The same exit-code semantics
apply (a failed `helm template ... > file` still trips `set -e`), but
the trace now only shows the commands themselves, not the rendered YAML.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
### Parallel post-install prep
LINSTOR / StorageClass / MetalLB setup, previously sequential `@test`
cases after the 15 m platform HR wait, now run as a background prep that
completes during the platform reconcile. None of them gates platform HR
reconcile, so they can move out of the critical path.
New helper `hack/e2e-post-install-prep.sh` polls for its prerequisites
and is awaited at the end of the `Install Cozystack` test so failures
still surface deterministically.
### T2-race fix
Wait on the LINSTOR HR being `Ready` before polling for
`deploy/linstor-controller`. The old 5-min hard timeout fired during
cold installs where the LINSTOR HR's dependency chain (cert-manager →
piraeus-operator-crds → piraeus-operator → linstor) takes longer than 5
min to resolve.
### E2E uses helmReleaseInterval=30s
E2E install passes `--set cozystackOperator.helmReleaseInterval=30s` to
eliminate the 8–10 min Flux dead zone where dependency-blocked HRs are
requeued only every 5 min. Operator-side flag and chart value landed in
#2509; production defaults are unchanged.
Surfaced from #2500.
### Release note
```
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
- Improved end-to-end test initialization to run concurrent post-install
preparation, wait for cluster release reconciliation, verify component
readiness, and validate automated storage configuration.
* **Chores**
- Added background post-install setup and logging to increase test
reliability and speed up cluster initialization and teardown
diagnostics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Replaces fragile `sleep N` / `timeout N until …; do sleep …; done`
patterns in `hack/*.bats` and `hack/*.sh` with `kubectl wait
--for=condition=…` plus a small existence backstop:
```sh
until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
kubectl wait <resource> --for=condition=Ready --timeout=…
```
### Why the existence backstop matters
`kubectl wait` against a not-yet-created resource sits silently for the
full timeout and then fails with `error: timed out waiting for the
condition` — useless for diagnosis. With the backstop, the test fails
fast with a clear "resource never appeared" error.
Existence backstop added at 14 sites in this sweep across:
- `redis`, `kafka`, `openbao`, `harbor`, `mongodb`, `qdrant`,
`clickhouse`, `mariadb`, `postgres`, `bucket`, `external-dns`,
`foundationdb` bats
- `hack/e2e-install-cozystack.bats`
- `hack/e2e-apps/run-kubernetes.sh`
- `hack/e2e-test-openapi.bats` (port-readiness wait replacing fixed
`sleep 5` after `kubectl proxy`)
The vm-disk hunk in `hack/e2e-apps/vminstance.bats` is included here;
the vm-instance hunks land separately via the `daniil/split-vminstance`
PR.
### Genuine sleeps annotated
The 3 remaining genuine sleeps (LINSTOR node membership propagation,
MetalLB end-to-end HTTP path, HR-count emission heuristic) are annotated
with `# TODO` + rationale so a future reader knows they are intentional.
Same green-path semantics; failures now surface in seconds instead of
after a 5-minute timeout. Surfaced from #2500.
### Release note
```
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Improved e2e stability by replacing fixed sleeps with timeout-based
polling across many integration tests, waiting for operator-created
resources to exist before readiness checks to reduce race conditions and
flakes.
* **Documentation**
* Added clarifying notes for LoadBalancer retry behavior and improved
local proxy readiness checks in e2e scripts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Two complementary changes that depend on the operator's
`--helmrelease-interval` flag landing via #2509:
1. Installer chart exposes `cozystackOperator.helmReleaseInterval`
(default empty -> flag not rendered -> operator default 5m
applies). E2E install passes `30s`. Eliminates the 8-10 min Flux
dead zone where dependency-blocked HRs (waiting on cert-manager
webhooks, CRDs) are requeued only every 5 min.
2. LINSTOR / StorageClass / MetalLB setup, previously sequential
`@test` cases after the 15 m platform HR wait, now run as a
background prep that completes during the platform reconcile.
None of them gates platform HR reconcile, so they can move out
of the critical path. Helper script `hack/e2e-post-install-prep.sh`
polls for its prerequisites and is awaited at the end of the
`Install Cozystack` test so failures still surface deterministically.
Includes the T2-race fix: wait on the LINSTOR HR being `Ready`
before polling for `deploy/linstor-controller`. The old 5-min hard
timeout fired during cold installs where the LINSTOR HR's dependency
chain takes longer than 5 min to resolve.
Production defaults are unchanged. The 30 s interval is opt-in per
install command.
Depends on:
- #2509 (operator-side `--helmrelease-interval` flag).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
## Why
The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:
- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.
This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.
## What
### New operator flags
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.
### Chart exposure
`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.
## Production safety
- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)
## Verification
- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`
## Followups (deliberately not bundled)
- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.
## Test plan
- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.
* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.
* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Why
helm v3 has a chicken-and-egg with charts that ship their own Namespace
resource:
- **With `--create-namespace`** on the install: helm pre-creates the
namespace via plain kubectl-create (no helm meta annotations); the
chart's own Namespace apply then fails with `already exists`.
- **Without `--create-namespace`**: helm fails immediately because it
can't write its release-secret to a non-existent namespace.
The cozy-installer chart shipped a `Namespace cozy-system` resource
since the chart's introduction (December 2025). Every cold install since
has hit this conflict. It was masked by the 3× retry on `Install
Cozystack` in `.github/workflows/pull-requests.yaml`: attempt 1 fails
with the conflict, attempt 2 sees the existing failed release and takes
the upgrade code path which patch-merges instead of strict-create.
Surfaced when retries on the install step were dropped.
## What
- Remove `Namespace cozy-system` from
`packages/core/installer/templates/cozystack-operator.yaml`.
- Add `packages/core/installer/templates/cozy-system-labels.yaml`: a
pre-install/pre-upgrade hook (ServiceAccount + ClusterRole +
ClusterRoleBinding + Job) that patches `cozystack.io/system=true` and
`pod-security.kubernetes.io/enforce=privileged` onto the namespace.
- Hook RBAC is scoped to `verbs: [get, patch]` on `resourceNames:
[cozy-system]` only.
- `hook-delete-policy: before-hook-creation,hook-succeeded` so the hook
surface only exists during install/upgrade.
Install commands now use `helm upgrade --install --namespace cozy-system
--create-namespace …` — standard pattern, matches kube-prometheus-stack
/ argo-cd / cert-manager.
## Verification
End-to-end on a kind cluster:
- Cold install: 3.3s clean, no retries needed
- Upgrade (re-run): 3.0s, idempotent, namespace stays labeled
- Helm uninstall + re-install: clean
## Test plan
- [ ] CI on this PR passes E2E
- [ ] Existing clusters: namespace already labeled correctly, hook
re-asserts on next upgrade — no migration needed
```release-note
fix(installer): cold installs no longer fail on the chart's Namespace resource colliding with `helm --create-namespace`; cozy-system is now bootstrapped via a pre-install hook.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an automated pre-install/pre-upgrade step that applies system
identity and pod-security labels to the system namespace during
deployment, run in a hardened one-shot task before normal workloads
start.
* **Chores**
* Separated namespace initialization from the operator install flow so
the operator no longer renders or retains the system namespace directly,
improving installation sequencing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- kafka.bats: use $name in the kafkas existence backstop, matching the
variable used in the kubectl wait below it (consistency with the rest
of this PR's backstops).
- e2e-install-cozystack.bats: the Tenant→HelmRelease materialisation is
asynchronous (cozystack-controller observes the Tenant CR and creates
the HR), so guard hr/tenant-test with the same existence backstop
pattern used elsewhere.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
With Strategy=RetryOnFailure + RetryInterval=30s plus the bats
--set cozystackOperator.helmReleaseInterval=30s override, the existing
2m wait converges without the manual flux reconcile --force band-aid.
Removing the workaround turns CI itself into the test of the fix.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used
by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks.
- Skip hostNetwork=true and not-ready/CNI tolerations on the hosted
variant: managed clusters provide CNI before install, and kube-system
may carry strict PSA there that would reject hostNetwork pods.
- Add packages/core/installer/templates/cozy-system-namespace.yaml gated
on .Values.bareNamespace (default false). The top-level Makefile's
manifests target now passes --set bareNamespace=true and renders
Namespace + operator into _out/assets/cozystack-operator-*.yaml so
the kubectl-apply install path stays self-contained. Helm
install/upgrade callers keep the default and bootstrap via
--create-namespace + the pre-install labeler hook.
- Add a bats assertion in hack/e2e-install-cozystack.bats checking
cozy-system carries pod-security.kubernetes.io/enforce=privileged and
cozystack.io/system=true after helm install.
- Inline comment near hostNetwork=true documenting why dnsPolicy left
at Default (cluster DNS not up at install time;
ClusterFirstWithHostNet would not help).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server)
fail when replicas start at materially different times due to
image-pull stagger across nodes. Add a DaemonSet-based pre-pull step
that runs before `helm install`, ensuring every node has the images
cached so replicas start within milliseconds of each other.
The script accepts image refs on stdin and creates one container per
image so pulls run in parallel — total time = max(any single image)
rather than sum. The bats test sources image refs directly from the
rendered platform charts via `yq`, walking only PodSpec-shaped
objects, so version bumps stay in sync without a hardcoded list.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Sweep across `hack/*.bats` and `hack/*.sh` replacing fragile
`sleep N` / `timeout N until …; do sleep …; done` patterns with
`kubectl wait --for=condition=…` plus a small existence backstop:
until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
kubectl wait <resource> --for=condition=Ready --timeout=…
Existence backstop added to 33 `kubectl wait` sites across 12 bats
files (redis, kafka, openbao, harbor, mongodb, qdrant, vminstance,
clickhouse, mariadb, postgres, bucket, install). Without it,
`kubectl wait` against a not-yet-created resource sits silently for
the full timeout; with it, the test fails fast with a clear
"resource never appeared" error.
OpenAPI tests' fixed `sleep 5` after starting `kubectl proxy` is now
a port-readiness wait. The 3 remaining genuine sleeps (LINSTOR node
membership propagation, MetalLB end-to-end HTTP path, HR-count
emission heuristic) are annotated with TODO + rationale so a future
reader knows they are intentional.
Same green-path semantics; failures now surface in seconds instead
of after a 5-minute timeout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).
E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.
Production defaults are unchanged. The override is opt-in per install.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Four follow-ups from review round 4:
1. BLOCKER: the Update(forceAllowCreate=true) path delegates to Create() when the object does not yet exist (rest.go:452) — the typical kubectl apply upsert flow. Add TestUpdate_ForceAllowCreate_RejectsTenantDashName using a fake client so a future refactor of that delegation cannot silently bypass the tenant name check that r.validateNameFormat alone cannot catch.
2. BLOCKER: the e2e BATS test used || true inside the command substitution, which swallowed the kubectl exit code. Rework the test to capture exit code and stdout+stderr explicitly, then assert the exit code is non-zero before asserting on the error message. This distinguishes validation-success (kubectl exit 0 — regression) from environmental failures (exit non-zero but wrong message) from the happy path.
3. Extract TenantKind = "Tenant" as a named constant in the validation package with a comment pointing at the upstream ApplicationDefinition source of truth, and switch the kindName check to use it.
4. Add a clarifying comment on TestValidateApplicationName_TenantLengthFallthrough that it pins an architectural layering decision and is not a user-facing requirement, so a future promotion of tenant length into tenant-specific wording is a legitimate change rather than a test regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three follow-ups from review:
1. Add defensive cleanup of the foo-bar tenant before and after the e2e regression test. If a prior run left the object in the cluster (e.g. a transient validation regression), the test no longer inherits that state; and if the test itself trips a regression in the future, the cluster is left clean for subsequent tests.
2. Switch kubectl --validate=false to --validate=ignore. The bool form was deprecated in kubectl 1.25 and only kept as an alias; --validate=ignore is the modern, stable spelling.
3. Drop the brittle Contains("63") assertion in TestValidateApplicationName_TenantLengthFallthrough. It pinned the test to the exact DNS-1035 error text from k8s.io/apimachinery, which could break on unrelated upstream wording changes. The surviving assertions still cover the intent: an error exists and it is not the tenant-specific message.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Follow-up from review round 2:
The BATS regression test now pins a namespace precondition, disables kubectl client-side validation with --validate=false so the request is guaranteed to reach the server-side name check, and documents the intent of each assertion. Previously a client-side schema rejection could have produced a false negative on the tenant-specific error grep, and a missing tenant-root namespace could have produced an unrelated failure that was hard to diagnose.
Also pin the edge case where a tenant name consists of valid characters but exceeds the DNS-1035 63-char label limit. The resulting error is intentionally the generic DNS-1035 message because length is not a tenant-specific constraint — the package-level function is not responsible for the stricter Helm-release-prefix length budget that REST.validateNameLength enforces.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two follow-ups from review:
1. The original tenantNameRegex ^[a-z0-9]+$ let leading-digit names like 123foo slip past the tenant check and receive a generic DNS-1035 error, which defeated the goal of surfacing a tenant-specific message. Require a leading lowercase letter so every tenant-invalid name returns the tenant-contract error.
2. Add a BATS regression test in hack/e2e-install-cozystack.bats that exercises the aggregated API end-to-end. Unit tests construct REST{kindName: "Tenant"} by hand, so a future change to ApplicationDefinition kind registration could break real behavior without breaking the unit tests.
Also pin the error-message contract with a new TestValidateApplicationName_TenantErrorMessage that asserts every tenant-invalid input returns a message containing 'tenant names must' (not the generic DNS-1035 text).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
On resource-constrained CI runners, HelmReleases may take close
to 15 minutes to reconcile, leaving very little time for CAPI
deployments to start and become available. Increase timeouts:
- CAPI deployment discovery: 60s -> 180s
- CAPI deployment availability: 1m -> 5m
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
Add external-dns as a standalone self-managed application in
`packages/extra/external-dns/`, allowing tenants to deploy and configure
their own DNS management directly from the dashboard.
## Motivation
Tenants need the ability to manage their own DNS domains with their own
provider. Following the [developers
guide](https://github.com/cozystack/website/pull/413), this is
implemented as an extra package (like `ingress` and `seaweedfs`) using
the HelmRelease-based pattern, rather than embedding it in the tenant
chart.
This enables multi-tenant scenarios where:
- Tenant A uses Cloudflare for `domain-a.com`
- Tenant B uses AWS Route53 for `domain-b.com`
- Each tenant deploys and manages external-dns independently from the
dashboard
## Changes
- **New package**: `packages/extra/external-dns/` — standalone
HelmRelease-based application
- **New PackageSource**:
`packages/core/platform/sources/external-dns-application.yaml` —
references `system/external-dns` and `extra/external-dns` components
- **Cleaned tenant chart**: removed the previously embedded
`externalDns` block from `packages/apps/tenant/`
## Features
- Support for 9 DNS providers: cloudflare, aws, azure, google,
digitalocean, linode, ovh, exoscale, godaddy
- Per-provider credential configuration with full JSON schema validation
- Domain filtering via `domainFilters`
- Configurable sync policy (`sync` or `upsert-only`)
- Namespaced operation (`namespaced: true`) for tenant isolation
- Unique `txtOwnerId` per namespace to prevent DNS record conflicts
- Resource sizing via presets or explicit CPU/memory
## Usage Example
Deploy from the dashboard, or via values:
```yaml
# Cloudflare
provider: cloudflare
domainFilters:
- example.com
cloudflare:
apiToken: "your-cloudflare-api-token"
```
```yaml
# AWS Route53
provider: aws
domainFilters:
- example.org
aws:
accessKeyId: "AKIAXXXXXXXX"
secretAccessKey: "your-secret-key"
region: "us-east-1"
```
## Test plan
- [ ] `helm template external-dns packages/extra/external-dns/ --set
provider=cloudflare --set cloudflare.apiToken=test` renders correctly
- [ ] `helm template external-dns packages/extra/external-dns/` fails
(provider required)
- [ ] `helm template wrong-name packages/extra/external-dns/ --set
provider=cloudflare` fails (release name check)
- [ ] Deploy external-dns from tenant dashboard
- [ ] Verify HelmRelease is created in tenant namespace with namespaced
RBAC
- [ ] Create an Ingress and verify DNS record is created
- [ ] Verify no conflict with global external-dns instance
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an External DNS package for automatic DNS record management.
* Support for 9 DNS providers: Cloudflare, AWS, Azure, Google,
DigitalOcean, Linode, OVH, Exoscale, GoDaddy.
* Helm-based deployment with namespaced/system variants and release
configuration options.
* Configurable synchronization policies, domain filtering, provider
credentials, extra args, and resource presets.
* **Documentation**
* New README and schema-driven values documentation for installation and
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The victoria-metrics-operator v0.68.1 renamed VMCluster status field
from .status.clusterStatus to .status.updateStatus.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
After migrating VictoriaLogs from VLogs to VLCluster, the e2e test
still waited for the old vlogs/generic resource which no longer exists.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
kubectl wait fails immediately with NotFound if the CRD does not exist
yet. The operator creates CRDs asynchronously on startup, so wrap the
wait in a retry loop that tolerates the initial absence.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Explicitly check error from parsePlatformSourceURL instead of relying on
the implicit guarantee that installPlatformSourceResource already checked
it. This prevents latent bugs if startup order is ever restructured.
Add wait for platform PackageSource existence in E2E test before creating
Package resource, preventing flaky failures when operator startup is slow.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
IvanHunters
