admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-04 19:17:20 +09:00
Code Issues Packages Projects Releases Wiki Activity
4,102 Commits 551 Branches 294 Tags
fix/e2e-postprep-timeout
Commit Graph
147 Commits
Author SHA1 Message Date
Ivan Okhotnikov d937a91a9a test(e2e): drop bash-only RETURN trap from bucket.bats port-forward 
The previous attempt added a bash `trap '...' RETURN` to clean up
kubectl port-forward when the test function returned. cozytest.sh
sources its converted bats test bodies into the runner shell, which
on the GitHub-hosted Linux runner is /bin/sh -> dash. dash does not
implement the bash-specific RETURN pseudo-signal, so the trap
command exits non-zero, the surrounding `set -eu` aborts the test
body immediately, and the captured output stops two lines after
`name=test` with no other diagnostic — the test never reaches the
mc S3 operations at all.

Drop the trap and rely on cozytest.sh's existing process-group
cleanup: when the test function returns, the leftover port-forward
job in the same group is terminated. This restores POSIX shell
compatibility while keeping the lifecycle fix from the previous
commit (no `bash -c '... &'` wrapper, port-forward stays alive
through the mc operations because it is a child of the calling
shell, not of a transient subshell that exits immediately).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-06-17 20:50:01 +05:00
Ivan Okhotnikov 862b19ffb6 test(e2e): keep bucket.bats port-forward alive across mc operations 
The test used to wrap kubectl port-forward in a transient subshell:

  bash -c 'timeout 100s kubectl port-forward ... > /dev/null 2>&1 &'

The wrapper subshell exits immediately after backgrounding the
kubectl process, the kubectl child is reparented to PID 1, and on
the GitHub-hosted runner it gets reaped between the `nc -z` readiness
probe and the first `mc cp` call. The captured failure looks like

  + bash -c timeout 100s kubectl port-forward ...
  + timeout 30 sh -ec until nc -z localhost 8333; do sleep 1; done
  + mc alias set rw-user https://127.0.0.1:8333 ... --insecure
  Added `rw-user` successfully.
  + mc cp --insecure /tmp/rw-test.txt rw-user/$BUCKET/rw-test.txt
  mc: <ERROR> Put "https://127.0.0.1:8333/$BUCKET/...":
    dial tcp 127.0.0.1:8333: connect: connection refused

even though every S3 resource (Bucket, BucketClaim, both BucketAccess)
is Ready/Granted=true at that point and the seaweedfs-s3 Service is
healthy in tenant-root.

Run kubectl port-forward as a job of the calling shell, capture its
PID, and tear it down via a bats RETURN trap. The trap is enough on
its own: the test no longer needs the outer `timeout 100s` belt
because the port-forward will be killed when the test function
returns, even on failure. Also switch the readiness probe to
127.0.0.1 to match the mc alias and avoid any IPv6 resolution path
on hosts where kubectl binds IPv4 only.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-06-17 20:50:00 +05:00
Ivan Okhotnikov 18f47f12e8 test(e2e): bind bucket.bats S3 client to 127.0.0.1 instead of localhost 
`kubectl port-forward service/seaweedfs-s3 -n tenant-root 8333:8333`
binds the listener to 127.0.0.1 only — it has no IPv6 listener. The
runner's getaddrinfo resolves `localhost` to `::1` first, and `mc`
keeps a single connection target across the whole alias, so the
initial `mc alias set` (which only sets credentials, no network I/O)
silently succeeds while the subsequent `mc cp` upload tries
`https://[::1]:8333` and gets immediate `connection refused`. The
test then exits with the misleading bucket-test failure even though
the S3 backend, credentials, and bucket all reconciled correctly.

Use a literal IPv4 address (`127.0.0.1`) for the mc aliases and the
`nc -z` readiness probe so the test no longer depends on the
runner's IPv4/IPv6 resolution policy.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-06-17 20:50:00 +05:00
Myasnikov Daniil 5b986e0a47 test(e2e): wait out nested-tenant teardown in gateway.bats 
The last three gateway tests create nested tenants (gwprop,
gwparent/gwchild, rparent/rchild) and deleted them fire-and-forget,
with parent and child deletes back-to-back. The file exits in ~25s
while the uninstalls run on for minutes: each tenant uninstall blocks
on a cleanup Job in cozy-system, and a parent deleted alongside its
child wedges on the child's still-terminating namespace.

On the tenants helm-controller shard (--concurrent=5) the leftovers
from one gateway.bats run — three stuck parent uninstalls, one stuck
child uninstall, and the rchild HR still mid-install — occupied
exactly 5 workers, so the next test in line (harbor) sat unreconciled
for its whole 5m HR-Ready budget and failed without ever erroring.

Sequence the teardown instead: delete child, wait for its HR to go
away, then delete parent and wait again. The waits are hard failures
on purpose — if a tenant uninstall genuinely wedges, gateway.bats is
the test that should go red, not whichever app test runs next.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-06 13:17:31 +05:00
myasnikovdaniil c435efa199 feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602) 
## What this PR does

Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.

Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).

Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.

## Commits

**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).

**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.

**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).

**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.

## Scope discipline

This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.

## Verification

- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.

### Release note

```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).

When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
  * New SeaweedFS DB Helm chart and optional managed DB release.

* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.

* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
  * Removed aggressive HelmRelease force-upgrade/install flags.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-02 14:08:39 +05:00
Aleksei Sviridkin 2d58479f06 feat(platform): add kuberture as optional system package (#2647) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional
system package. Kuberture is a small controller that translates the
`default/kubernetes` EndpointSlice into headless Services with
external-dns-style annotations, closing the gap that external-dns cannot
use EndpointSlice as a source directly:

```text
EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS
```

The package is opt-in via `bundles.enabledPackages:
[cozystack.kuberture]`. Operators populate `config.outputs` per
HelmRelease — each output targets a specific external-dns instance by
its `annotationPrefix`, so a single kuberture install can serve multiple
external-dns instances (e.g. one publishing to the public zone, another
to an internal corporate prefix) simultaneously.

The vendored chart at `packages/system/kuberture/charts/kuberture/` is
pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and
pinned by both version AND OCI manifest digest in the Makefile
(mirroring the ouroboros lockstep pattern). The controller image at
`ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim
values.yaml. Both pins advance in lockstep with each upstream release;
the procedure is documented inline.

Install-order dependencies declared in `PackageSource`:
`cozystack.networking` (CNI before pod scheduling) and
`cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the
chart's enabled-by-default ServiceMonitor manifest). External-dns is
deliberately NOT a dependency — kuberture creates annotated Services
regardless of whether external-dns is installed; external-dns is a
downstream consumer, not a runtime prerequisite.

Coverage:

- **helm-unittest**
(`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases):
fail-fast on empty outputs, ServiceMonitor enabled by default, source
pinned to `default/kubernetes`, multi-output prefix isolation,
empty-string `annotationPrefix` schema rejection, omitted-prefix
fallback to controller default, ConfigMap → Deployment volume mount,
controller image pinned by tag+sha256 digest, Role permits full Service
lifecycle, metrics containerPort derived from `metricsBindAddress`.
- **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package
cozystack.kuberture` with two outputs (different `annotationPrefix`
values), waits for HelmRelease and controller readiness, asserts both
output Services are headless and carry the right
`hostname`/`target`/`ttl` annotations under their configured prefix,
cross-checks per-output isolation (each Service only carries its own
prefix's annotations).

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not applicable — no UI changes.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Adds kuberture as an optional system package to publish the Kubernetes
API endpoint to DNS via configurable external-dns outputs; includes Helm
chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart
values/schema.

* **Tests**
* Adds Helm unit tests and an end-to-end smoke test validating
controller readiness, headless Service annotations, and external-dns
routing/probe behavior.

* **Documentation**
* Adds a full README with usage, configuration examples, and
vendoring/pinning guidance.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---

Documentation companion:
[cozystack/website#539](https://github.com/cozystack/website/pull/539).
 2026-06-02 09:40:26 +03:00
Myasnikov Daniil a3a7a6d516 test(e2e): add HR existence backstop for external-dns ${name}-system 
The two external-dns bats tests already poll for the user-facing ${name}
HelmRelease before calling kubectl wait, but they immediately wait on the
${name}-system HR without the same backstop. The system HR is materialised
asynchronously by the controller, so kubectl wait can race and exit non-zero
when the object does not exist yet. Apply the same existence poll pattern
to both occurrences.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-28 21:25:25 +05:00
Myasnikov Daniil 401993ac62 test(e2e): wait for parent HR Ready before downstream asserts 
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape:
the bats steps assumed a side-effect ordering ("by the time X is true,
the parent HR's helm install is done") that held under v2.7's faster
dispatch but breaks under v2.8 where helm-controller walks kstatus on
every applied resource before flipping the HR Ready.

run-kubernetes.sh:
  After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx)
  reaches Ready, the remediation guard reads `.status.history` on the
  parent kubernetes-${test_name} HR. The parent's helm install can still
  be Reconciling at that point because kstatus is finishing its full
  sweep; `.status.history` is empty until install completes, so the
  guard exits 1 with "empty .status.history on a Ready HR". kubernetes-
  latest masked this because the ouroboros assertions add ~10m of
  buffer; kubernetes-previous (no ouroboros) hit the guard with the
  parent still installing. Add an explicit
    kubectl wait hr/kubernetes-${test_name} --for=condition=ready
  before the guard.

vminstance.bats:
  The test waited 120s for `kubectl get vmi vm-instance-$name` to
  return a VMI with an assigned IP, then waited for the parent HR Ready
  separately afterwards. Under v2.7 the parent HR's helm install
  applied the VirtualMachine CR fast enough that kubevirt had created
  the VMI before the 120s budget expired. Under v2.8 the dispatch is
  slower and the VMI poll returned NotFound for its entire budget.
  Reorder: wait for HR Ready first (which implies the VirtualMachine
  CR is applied + kstatus-ready), then poll for the VMI with IP.
  Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c664b868bc)
 2026-05-28 12:43:50 +05:00
Myasnikov Daniil 7b9f286faa test(e2e): bump app HR-Ready waits for Flux v2.8 kstatus 
Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.

Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.

Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.

Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.

Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5ef02ee7b3)
 2026-05-28 12:43:50 +05:00
Myasnikov Daniil ea7a8c8675 test(e2e): quote HelmRelease names in kubectl wait 
Both \${test_name} and \${component} are literal-safe today (no
whitespace, no glob characters), so this is a style-only change — but
quoting variable expansions in kubectl arguments matches the rest of
the file and removes a future-foot-gun if a test name ever grows a
hyphen-y suffix that interacts oddly with completion or globbing.

Scoped to the two HelmRelease wait lines flagged by CodeRabbit and
echoed by the human reviewer; the machinedeployment wait on the
preceding line keeps its existing style.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-27 20:36:10 +05:00
Myasnikov Daniil 64c443caf1 test(e2e): split local k8s_version declaration from yq invocation 
\`local k8s_version=\$(yq ...)\` masks the exit status of the command
substitution — the assignment to a local variable always succeeds, so
a failing yq is swallowed. Declare the local first, then assign, so
non-zero yq exits propagate via shell errexit semantics if the script
ever runs under \`set -e\`.

Pure shell hygiene; no behavioral change on the success path. Surfaced
by CodeRabbit and confirmed by the human reviewer as non-blocking but
worth applying.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-27 20:36:10 +05:00
Myasnikov Daniil 8c6e149177 test(e2e): delete standalone ouroboros.bats now folded into kubernetes-latest 
The previous commit (3c81261b) folded the ouroboros hairpin-NAT
reconciliation assertions into the kubernetes-latest tenant under
run-kubernetes.sh, but did not remove the standalone bats file.

The e2e workflow auto-discovers every hack/e2e-apps/*.bats and dispatches
test-apps-<name> via the wildcard rule in packages/core/testing/Makefile.
Leaving ouroboros.bats in the tree meant the assertions ran twice: once
on the standalone ~25m Kamaji-backed tenant and again as the folded
addon on the kubernetes-latest tenant — the opposite of the consolidation
the fold was intended to deliver.

The folded assertions in hack/e2e-apps/run-kubernetes.sh cover the same
surface (HR Ready, controller pod Running, coredns-custom rewrite-line
injection, and end-to-end DNS resolution from inside the tenant), so
the standalone file is dead weight after the fold.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-27 20:36:09 +05:00
Myasnikov Daniil 68a9a6e5ad test(e2e): bump NFS pod wait to 10m to fit slow-path chain 
Previous run failed kubernetes-latest (and then kubernetes-previous via
cascade) on the 5m budget for nfs-test-pod to reach Succeeded. The
chain is host linstor prime PVC + CDI import + tenant CSI attach +
busybox image pull + containerd CreateContainer (which occasionally
stalls past kubelet's 2m CRI deadline and retries with name-reservation
conflict). ~40s on the happy path, 6m+ in observed worst case.

When kubernetes-latest times out, its tenant cluster (2x 20Gi worker
PVCs) leaks because set -e exits before any cleanup. kubernetes-
previous then runs, adds another 40Gi, and the host tenant-quota
(100Gi) is exhausted -- every downstream app test (mariadb, mongo,
postgres, qdrant, redis, openbao, vminstance) fails with
ErrCreatingPVCPrime / exceeded quota.

Bumping the wait to 10m gives the slow path room. Headroom costs
nothing on green runs since the pod is busybox echo.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c5cc7b60b8)
 2026-05-27 20:36:09 +05:00
Myasnikov Daniil 04a54b01cf test(e2e): wait for tenant CSI HR Ready before NFS test 
kubernetes-previous fails on the NFS pod step. From cozyreport + job log:

  - Pod scheduled to tenant worker
  - 7x FailedAttachVolume over 35s: CSINode does not contain driver
    csi.kubevirt.io (kubevirt-csi-node DaemonSet still rolling out)
  - Attach succeeds at T+1m08s after the driver registers
  - busybox pull queued ~1m25s, completes at T+2m48s
  - CreateContainer in containerd then stalls past kubelet's CRI
    deadline; retry trips "failed to reserve container name ... is
    reserved for <prev>" -- the 5m pod-Succeeded wait then expires.

Root cause: run-kubernetes.sh creates the NFS PVC+Pod before asserting
kubernetes-${test_name}-csi is Ready. That HR lives in the parent
cluster but installs cozy-kubevirt-csi-node into the tenant via
kubeConfig.secretRef; helm-controller walks kstatus on the tenant
DaemonSet using the tenant kubeconfig. HR Ready in parent therefore
implies kubevirt-csi-node Pods Ready on every tenant worker, which
implies csi.kubevirt.io is published in CSINode. The existing wait at
line 354 runs AFTER the NFS test, so it does not gate the race.

Move the wait earlier with a 10m budget. Post-NFS sibling waits stay
as is, so component coverage is unchanged.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 394b1b2b9a)
 2026-05-27 20:36:09 +05:00
Myasnikov Daniil 5035ac1414 test(e2e): fold ouroboros assertions into kubernetes-latest 
The standalone hack/e2e-apps/ouroboros.bats provisioned a full Kamaji-
backed tenant Kubernetes cluster (control plane + 2 CAPI workers + base
addons) just to flip `addons.ouroboros.enabled=true` on top, then ran
~15m of hairpin-NAT reconciliation assertions. Cluster shape was
identical to what hack/e2e-apps/run-kubernetes.sh already provisions for
kubernetes-latest, so the second Kamaji bringup was pure duplication —
roughly 25-45m of CI wall-time per run.

Add an `enable_ouroboros` 4th parameter to `run_kubernetes_test()`. When
"true", inject the ouroboros addon block into the Kubernetes CR's
addons map and run the post-cluster assertions inline:

- Wait for the ouroboros HR Ready in the host namespace.
- Verify coredns-custom ConfigMap exists in tenant kube-system.
- Wait for the ouroboros controller pod Ready.
- Apply an Ingress with a TLS host in tenant default namespace.
- Poll coredns coredns-custom for the rewrite line (5m budget).
- Resolve the hairpin host from inside the tenant via a netshoot pod
  and assert it points at the ouroboros-proxy ClusterIP.
- Clean up dnscheck pod + Ingress.

kubernetes-latest passes "true" (gets ouroboros coverage); kubernetes-
previous leaves it empty (control-plane-only smoke remains).

(The standalone ouroboros.bats was removed in the preceding commit
alongside an unrelated foundationdb chart fix — the git rm staging had
already happened when the foundationdb commit landed.)

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit b5c9c2ac90)
 2026-05-27 20:36:09 +05:00
Aleksei Sviridkin c85e473523 test(e2e): inheritance attach asserts controller writes, not Cilium Accepted 
The previous e2e check waited for
`HTTPRoute.status.parents[0].conditions[Accepted]=True` after
applying a child tenant's HTTPRoute against the parent's Gateway.
That condition is set by Cilium after the listener's TLS Secret
becomes Ready — and in the e2e cluster every fresh certificate fails
to issue (LE prod refuses every `.example.org` hostname by policy,
visible across alerta/grafana/dashboard/seaweedfs-s3 cluster certs
in cozyreport). So Cilium never set Accepted, the route timed out
3/3 retries, and the failure was environmental — not a regression
in the inheritance code path.

The contract the test exists to pin is cozystack-controller's
behaviour, which has two observable writes:

1. The parent Gateway grows a per-listener HTTPS entry for the
   child route's hostname (proves collectHostnameClaims saw the
   route through the inheritance label and reconcileGateway
   appended the listener).
2. A per-listener Certificate object appears in the parent
   namespace with the route's hostname in spec.dnsNames (proves
   reconcilePerListenerCertificates ran for the same hostname).

Both writes are deterministic and depend only on the inheritance
label + the HTTPRoute existing — not on ACME, not on Cilium TLS
readiness. The test now polls those two objects directly within
shorter, scenario-appropriate timeouts (120s for the listener, 60s
for the Certificate). The route Accepted check is dropped with an
explanation in the test comment so a future cluster with a working
ACME doesn't get a misleading pre-condition added back. The
namespace inheritance label is also verified up front as the
precondition collectHostnameClaims's filter reads.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-25 23:51:00 +03:00
Aleksei Sviridkin 4120f8795e test(e2e): cover child-tenant inheritance attach flow 
Replaces the now-obsolete "Package admission rejects tenant-* in
gateway.attachedNamespaces" e2e (the VAP it pinned was dropped in
the inheritance refactor) and adds two new e2e cases that cover
the end-to-end inheritance contract:

* "Package admission accepts gateway.attachedNamespaces with
  tenant-* entries" — regression guard against a future refactor
  re-introducing the gate; the hijack surface is now handled by
  Layers 4/5/7 (Tenant.spec.host VAP, namespace label immutability
  VAP, HTTPRoute hostname VAP), not by entries in attachedNamespaces.

* "child tenant without explicit gateway inherits _namespace.gateway
  from a Gateway-owning parent" — chart-level lockstep: a parent
  tenant with gateway=true produces both the namespace label and the
  cozystack-values Secret pointing at itself, and a child Tenant
  under that parent with the gateway field unset receives the
  parent's name in both places. Plus a negative assertion that the
  child does NOT get its own gateway HelmRelease.

* "child tenant's HTTPRoute attaches to parent's Gateway via
  inheritance label" — full cross-namespace attach: parent owns
  the Gateway, child inherits, an HTTPRoute in the child namespace
  pointing at parent's Gateway reaches Accepted=True once Cilium
  evaluates the label-based allowedRoutes selector. Route hostname
  uses the child's derived apex so Layer 7 admits it.

The Gateway-Programmed wait is intentionally omitted from the
last test — Programmed depends on the LB allocator that ships with
the e2e cluster, while Accepted is a pure spec evaluation on the
parent Gateway's allowedRoutes selector and is the property the
inheritance refactor is meant to deliver.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-25 19:07:40 +03:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium 
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-25 17:07:02 +03:00
Arsolitt 2086f3e6b6 test(kafka): assert ZooKeeper PVC-1 is created before deletion check 
Address review feedback from coderabbitai on hack/e2e-apps/kafka.bats:58:
The post-delete `while kubectl get pvc data-kafka-$name-zookeeper-1` loop
exits immediately when the PVC has never been created, so a regression
that prevents Strimzi from provisioning the second ZooKeeper PVC would
silently pass the reclamation check. Mirror the PVC-0 readiness guard
(existence + Bound) for PVC-1 before deleting the Kafka CR so the test
only proceeds once both PVCs are observed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-05-20 14:03:29 +03:00
Arsolitt c4b0606e67 test(kafka): verify ZooKeeper PVCs are reclaimed by the operator 
Address review feedback from gemini-code-assist on
hack/e2e-apps/kafka.bats:53-55:
Replace the manual kubectl delete of the two ZooKeeper PVCs with a poll
that waits for the Strimzi operator to reclaim them. With deleteClaim:
true on the ZooKeeper persistent-claim storage, the operator removes
the PVCs as part of Kafka CR deletion; manual deletion would either
race the operator, fail with NotFound, or silently mask a regression
in the reclamation path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-05-20 13:45:07 +03:00
Aleksei Sviridkin 5c188fede3 test(kuberture): cover platform install and multi-output annotations in e2e 
Smoke test that drives the cozystack.kuberture Package end-to-end on the
e2e cluster:

- creates a Package with two outputs targeting distinct annotationPrefix
  values so a single kuberture install can serve multiple external-dns
  instances simultaneously
- waits for namespace, HelmRelease and controller Deployment readiness
- asserts both output Services are headless and carry the right
  hostname/target/ttl annotations under their configured prefix
- cross-checks that each Service only carries its own prefix's
  annotations, not the other output's

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-15 14:03:32 +03:00
Aleksei Sviridkin 6f83d9a794 feat(ouroboros): bump to v0.7.2 with debug logging on tenants 
The v0.7.2 ouroboros release adds --log-level / OUROBOROS_CONTROLLER_LOG_LEVEL,
plants Debug logs at every informer event handler (AddFunc/UpdateFunc/
DeleteFunc with kind+namespace/name via describeObj) and at reconcile
entry/exit, wires SetWatchErrorHandler on Ingress / Gateway / HTTPRoute
informers, and surfaces cache-sync elapsed at Info. The diagnostic
surface targets the kamaji-tenant-via-konnectivity flake where bats
sees a fresh Ingress create succeed against the tenant apiserver but
the controller's coredns-custom rewrite lands roughly 7-10 minutes
later — without per-event visibility the failure mode is
indistinguishable between 'watch never fires', 'reconcile silently
errors', and 'API list returns empty until reflector reconnect'.

Bumps the chart-manifest digest, the controller-image digest, and the
cozystack-side image-pin assertion in lockstep. The bats deadline for
the coredns-custom rewrite check rises from 180s to 900s — even on
the broken-watch path the reflector eventually reconnects within
that budget, so the test stops failing on the symptom while we keep
the diagnostic on to learn what the symptom actually is.

cozystack tenants force controller.logLevel: debug via the
defaultOuroborosValues helper; operators that want quieter logs can
override through addons.ouroboros.valuesOverride. Two helm-unittest
fixtures pin both the default and the override path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-11 02:20:34 +03:00
Aleksei Sviridkin d6bcb27c53 test(ouroboros): dump tenant state on every assertion failure, not just HR wait 
The CI run after pinning controller.clusterDomain showed the test was finally getting past the helm-controller wait — the proxy /readyz unblocked, the HR reaches Ready, and the controller pod becomes Ready — but the next assertion (the rewrite snippet appearing in kube-system/coredns-custom within 180s of creating the probe Ingress) is still failing on every retry. Without a dump on this assertion the failure surfaces only as `❌ Test failed (exit 1)` with the polled `snippet=` empty lines preceding it; the controller logs that would explain why the reconciler skipped the Ingress are nowhere visible.

Refactor the existing inline diagnostic block into a `dump_tenant_state` helper that is invoked on every assertion's failure path. Wrap the rewrite-snippet check, the proxy ClusterIP non-empty check, and the dnscheck pod-phase check in `if ! ...; then dump_tenant_state; exit 1; fi` so each failure dumps the same set of host/tenant pods, events, controller and proxy logs, ingress-nginx Service+endpoints, and tenant CoreDNS Corefile that the original HR-wait failure path already covered. Also append a `kubectl get ingress -n default -o yaml` dump of the probe Ingress so a controller log line like "skipped Ingress without TLS host" is correlatable against the actual rendered manifest at the time of failure.

The follow-up fix for whatever the next CI run surfaces will land on this same branch.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-08 12:49:05 +03:00
Aleksei Sviridkin 89aab4459b test(ouroboros): port-forward + super-admin.conf for tenant kubectl path 
Every tenant-side kubectl call in hack/e2e-apps/ouroboros.bats was failing inside the e2e sandbox container with `dial tcp: lookup kubernetes-${cluster}.${ns}.svc on 169.254.169.254:53: no such host`. The bats test pulled the `super-admin.svc` variant of the tenant admin-kubeconfig, which points at the in-cluster Service DNS name `<release>.<ns>.svc:6443` — a name only host-cluster CoreDNS can resolve. The sandbox container's resolver is the Docker/QEMU host link-local 169.254.169.254, not the host-cluster CoreDNS, so every `KUBECONFIG=$kubeconfig kubectl ...` invocation in the test was returning NXDOMAIN.

The check that gated whether the test even reached its assertions (the `kubectl wait helmrelease kubernetes-${cluster}-ouroboros`) ran against the host kube-context, which masked the broken kubeconfig path until the helmrelease itself failed to reconcile and the diagnostic block fired — at which point every dump command returned only NXDOMAIN errors instead of actionable tenant-cluster state.

Switch to `super-admin.conf` (the operator-facing kubeconfig variant), rewrite `clusters[0].cluster.server` to `https://localhost:${pf_port}`, and run a `kubectl port-forward service/kubernetes-${cluster} ${pf_port}:6443` in the background — the same pattern hack/e2e-apps/run-kubernetes.sh already uses for tenant-cluster reachability. Replace the inline trap RM with a `cleanup_kubeconfig` helper that also kills the port-forward process so retries don't leak listeners.

This is a test infrastructure fix; the underlying ouroboros HelmRelease install timeout itself is a separate root cause that the next CI run, with usable diagnostics, will surface.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-07 23:39:29 +03:00
Aleksei Sviridkin c99be232ad test(ouroboros): dump tenant-side state on bats wait failure 
The smoke test at hack/e2e-apps/ouroboros.bats waits for the kubernetes-${cluster}-ouroboros HelmRelease to reach Ready and exits with `wait: timed out` on failure. cozyreport collects host-cluster state only — a tenant-side proxy or controller failure surfaces as opaque "0 of 2 pods ready" with no actionable signal, so every retry is fully blind.

Extract the tenant admin-kubeconfig before the wait (gated on the parent kubernetes HR being Ready, which is the precondition for the secret existing) and, on wait failure, dump the host HelmRelease describe, cozy-ouroboros pods, pod descriptions, recent events, controller and proxy logs (200 lines each), the upstream cozy-ingress-nginx Service and endpoints used as the proxy backend, and the tenant kube-system Corefile plus the coredns-custom ConfigMap data.

Drops the ouroboros-HR wait timeout from 25m to 20m to leave room for the diagnostic block within the bats test budget.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-07 20:20:33 +03:00
Andrei Kvapil 45e21a330f feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561) 
## What this PR does

Adds
[`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0)
— a Go reimplementation of
[`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy)
— and wires it into cozystack as a *conditional default*: a single
`publishing.proxyProtocol: true` switch turns PROXY-protocol on at the
host ingress-nginx **and** auto-deploys ouroboros to fix the resulting
hairpin-NAT problem (intra-cluster traffic to the cluster's own public
hostnames arriving at ingress-nginx without the PROXY header it now
requires). Default stays `false` — clusters that do not run
PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour
change.

The chart is platform-agnostic at v0.7.0: the controller and proxy
binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf`
inside the pod and compose the rewrite target FQDN and the backend host
at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on
`cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`,
federations on `k8s.example.com`, custom-domain RKE2/k3s — all work
without operator override. Verified end-to-end on a `cozy.local`
cluster: controller logs `cluster-domain=cozy.local` at startup, chart
emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked
`--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END
markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy
ClusterIP. Operators that prefer a chart-baked FQDN can still set
`controller.clusterDomain` explicitly — the explicit-override path is
preserved for non-Service backends and for split-horizon DNS topologies.

The chart runs in `coredns` mode for the host (the host CoreDNS is
Talos-managed and does not carry an `import` directive, so the
alternative would silently no-op there) and in `coredns-import` mode for
tenant deployments (tenants run cozystack-coredns, which carries the
import directive and ships an empty `kube-system/coredns-custom`
ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the
`ouroboros.override` data key without ever touching the chart-rendered
Corefile, so it does not race the chart that re-templates
`data.Corefile` on every Flux reconcile).

### Layout

* `packages/system/ouroboros/` — vendored chart from
`oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults
pinned in `values.yaml`: `controller.mode=coredns` (host scenario),
`proxy.target.namespace=cozy-ingress-nginx` (override the chart-default
`ingress-nginx` namespace; chart-default
`serviceName=ingress-nginx-controller` matches as-is), Gateway-API
watching left at upstream's default of `false`. `proxy.target.host` is
left empty so the proxy composes the backend host at runtime. Image is
digest-pinned at
`0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c`
and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is
intentionally not mirrored under `ghcr.io/cozystack/*`. The chart
manifest itself is also digest-pinned via
`OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e`
in the package Makefile, with a Chart.yaml-version vs. Makefile-version
assert that aborts the update target on lockstep drift. Helm-unittest in
`tests/` covers controller args, RBAC narrowing per mode, Gateway-API
gating, runtime-composition flag emission in both `coredns` and
`coredns-import` modes, image pin drift.
* `packages/system/coredns/templates/coredns-custom.yaml` +
`values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed
`import` prerequisite, plus the `import /etc/coredns/custom/*.override`
plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts`
mount at `/etc/coredns/custom/`. Same package serves host and tenant
CoreDNS, so this covers both layers without duplication. Helm-unittest
pins the rendered Corefile via a snapshot so future `make update` does
not silently drop the import directive. The empty rendered ConfigMap
deliberately omits `data:` — Helm three-way merge has no template-side
keys to compare against, so apiserver-side runtime writes (the
`ouroboros.override` data key) survive every chart upgrade. The
`notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the
invariant.
* `packages/core/platform/sources/ouroboros.yaml` — new
`cozystack.ouroboros` PackageSource for the host install (`dependsOn:
cozystack.networking, cozystack.ingress-application`).
* `packages/core/platform/sources/kubernetes-application.yaml` —
`kubernetes-ouroboros` component registered against `system/ouroboros`
so the tenant HelmRelease can reference it.
* `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` —
tenant-side HelmRelease deployed via the tenant admin-kubeconfig,
mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues`
helper flips `mode` back to `coredns-import` for tenants and mirrors
per-tenant `addons.gatewayAPI.enabled` into the controller (forcing
`--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the
controller on `WaitForCacheSync`). `controller.clusterDomain` is
intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from
`/etc/resolv.conf` and composes the rewrite target at runtime, so the
same wrapper renders correctly on `cluster.local`, `cozy.local`, or any
custom tenant cluster-domain. Operators on a non-default cluster-domain
can still override via
`valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a
conditional `gateway-api-crds` entry on the same gate. If
`addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled`
is false, the HelmRelease fails the render with a clear error rather
than silently skipping — the hairpin fix has nothing to fix against
without ingress-nginx, and a no-op addon that quietly does nothing is
harder to debug than an explicit render-time error. The valuesOverride
gateway-api guard accepts every truthy spelling Helm/YAML emit (bool
`true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML
`yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via
`toString`-then-`lower`-then-string-compare; direct bool `eq` would
error on type mismatch when an operator passes the value as a string
from `--set`, and case-insensitive folding catches `--set ...=On` /
`--set ...=True` that pure-lowercase comparison would miss. Tenant
disable is single-step: flipping `addons.ouroboros.enabled: false`
deletes the rendered HelmRelease, helm-controller runs `helm uninstall`
on the tenant, and the chart's pre-delete cleanup hook
(`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the
`ouroboros.override` key on the tenant's `kube-system/coredns-custom`
ConfigMap automatically. No render-time guard is wired on the tenant
side — gating on a `lookup` for the leftover HelmRelease would deadlock
the routine disable (the parent render runs before helm-controller
applies the missing-child diff, so the lookup would always find the
leftover at the moment of the flag flip). If the chart's hook fails to
land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job
timeout), operators recover manually via the documented tenant cleanup
recipe.
* `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef
with `enabled` + `valuesOverride`. `make generate` re-emitted README,
JSON schema, the Go struct + deepcopy, and the kubernetes-rd
ApplicationDefinition openAPISchema in lockstep.
* `packages/core/platform/values.yaml` — new `publishing.proxyProtocol`
flag with extensive comment covering the upstream-LB precondition (the
L4 LB in front of ingress-nginx must already be injecting PROXY-v1
headers before the flag flips on, otherwise external traffic breaks),
the host-vs-tenant fan-out (the flag does not cascade into tenants), the
Talos re-render flap window, and the asymmetric disable path. The host
disable path: flipping back to `false` reverts ingress-nginx
PROXY-protocol config (flip the LB off first), but the host ouroboros
itself stays installed because its Package CR carries
`helm.sh/resource-policy: keep` — an explicit `kubectl delete
package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages`
entry) actually triggers helm uninstall, at which point the chart's
pre-delete cleanup hook fires and patches the host Corefile
automatically. The values comment ships a manual `sed`-based cleanup
recipe as a fallback for the rare case where the chart's hook fails to
land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook
timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off
a `lookup` against the leftover Package CR to catch operators who flip
the flag back without confirming the cleanup landed.
* `packages/core/platform/templates/bundles/system.yaml` — flips between
the default-form and explicit-form `cozystack.ingress-application`
Package emission so the host ingress-nginx gets `use-proxy-protocol` +
`real-ip-header=proxy_protocol` injected only when the flag is true
(`use-forwarded-headers` and `compute-full-forwarded-for` are
deliberately *not* on the list — they would let any upstream proxy spoof
`X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`.
Host Gateway-API watching is left at the upstream chart's default of
`false` because cozystack does not install Gateway-API CRDs cluster-wide
on the host.
* `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the
tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in
tenant, controller pod running, and after creating a TLS Ingress the
rewrite snippet shows up in the import ConfigMap; an in-tenant `dig`
from a netshoot pod also confirms CoreDNS actually serves the rewrite —
pod-readiness alone passes vacuously since the upstream chart ships no
readiness probes).

### Why not KEP-1860 `ipMode=Proxy`

Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB
frontend entirely when `ipMode != VIP`, breaking Service ingress for
L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters
fronted by an external CCM-managed proxy LB. ouroboros is the correct
fix for L2/BGP-announce topologies that cozystack ships by default.

### Coverage

Covered by helm-unittest (host chart, coredns wrapper, tenant addon —
including the `kubernetesProvider` mock that exercises the lookup-driven
`proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil
CI/dry-run path, the gateway-api truthy-spelling acceptance across
`true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that
`--proxy-service-name` + `--proxy-service-namespace` are emitted instead
of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import`
modes, the assertion that `--target-service-name` +
`--target-service-namespace` are emitted for the proxy backend, and the
`_namespace.etcd`-empty render gate) and a tenant e2e bats test that
creates a real Ingress, asserts the rewrite snippet lands in the import
ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry
loop tolerating CoreDNS reload latency). The host-level
`publishing.proxyProtocol: true` path is not e2e-tested in this PR — it
requires a PROXY-protocol-aware upstream LB, which the cozystack e2e
harness does not stand up. That path is exercised manually and by docs
review.

End-to-end runtime auto-detect was verified manually on a `cozy.local`
cluster (dev17): chart deployed at v0.7.0, controller logged
`cluster-domain=cozy.local`, the rewrite block landed in
`kube-system/coredns` Corefile pointing at
`test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig
<ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first
attempt, and the chart's pre-delete hook restored the Corefile cleanly
on `helm uninstall`.

### Supply-chain note

Image and chart are pulled from a contributor's personal namespace
(`ghcr.io/lexfrei/...`). Existing precedent
(`packages/system/etcd-operator/Makefile` pulls from
`ghcr.io/aenix-io/...`) is similar but `aenix-io` is the
cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer
call.

Pairs with cozystack/website#527 (operator docs).

### Release note

```release-note
feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with
configurable enablement and tenant overrides.
* Added a packaged Ouroboros chart providing proxy, controller, CoreDNS
integration, external‑DNS and etc‑hosts modes.
* Introduced PROXY‑protocol gating with an explicit acknowledgement
flag.

* **Documentation**
* Expanded addon reference and platform docs with detailed
Ouroboros/PROXY‑protocol configuration and defaults.

* **Tests**
* Added extensive unit and e2e tests covering rendering, RBAC, uninstall
cleanup, and DNS behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-07 18:41:13 +02:00
myasnikovdaniil 32e75ab183 Merge branch 'main' into daniil/split-vminstance 
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com>
 2026-05-06 18:15:08 +05:00
Myasnikov Daniil 209fa08737 test(e2e): decouple vminstance test from vmdisk test 
"Create a VM Instance" referenced the VMDisk created by "Create a VM
Disk" via the shared name "test". This implicit ordering meant any
defensive pre-clean of vmdisks in test 2 would destroy the disk test 1
just provisioned, and the vm-instance chart's lookup-or-fail on the
DataVolume would then surface as "Specified disk not exists in cluster:
test".

Make each test self-contained:
- "Create a VM Disk" cleans up its own VMDisk at the end.
- "Create a VM Instance" creates its own VMDisk inline (renamed to
  "vmi-test" to avoid colliding with any in-flight finalizer drain on
  test 1's resources) and waits HR -> DV -> PVC before applying the
  VMInstance.

Replaces the pre-delete approach from the prior followup commit, which
broke the data dependency.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-06 08:59:42 +05:00
Aleksei Sviridkin 585ea7e089 test(e2e): smoke-test the ouroboros tenant addon 
Auto-discovered hack/e2e-apps/ouroboros.bats spins up a tenant
Kubernetes CR with addons.ingressNginx + addons.ouroboros enabled
and asserts the cozystack-side wiring delivers a working hairpin
fix end-to-end:

  * the cozy-ouroboros HelmRelease in the tenant's host namespace
    reaches Ready (chartRef + admin-kubeconfig wiring is correct);
  * the cozystack-coredns wrapper rendered the kube-system/coredns-custom
    ConfigMap inside the tenant (the import-glob prerequisite is in
    place);
  * the controller pod is running in cozy-ouroboros inside the tenant;
  * after creating an Ingress with a TLS hostname, the tenant's
    coredns-custom ConfigMap eventually carries a `rewrite name <host>`
    line under the ouroboros.override key — proves the controller is
    reconciling, not merely Pod-Ready (the upstream chart ships no
    readiness probes, so a controller stuck on WaitForCacheSync would
    pass a pod-readiness check vacuously; the rewrite assertion catches
    every cozystack-specific failure mode the unit tests cannot — wrong
    namespace, missing CRDs, narrowed RBAC, gateway-api gate misfires).

End-to-end PROXY-protocol header roundtrip (the actual hairpin fix
in action) is covered by the upstream chart's own e2e on every push;
this file is the cozystack-side integration test that the cozystack
glue (chart vendoring, kubernetes-application registration, tenant
HelmRelease wiring, gateway-api gating, coredns-custom prerequisite)
holds together.

cozystack hack/cozytest.sh is plain shell, not real bats, so kubectl
output is captured directly without the `run` helper.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-06 06:48:07 +03:00
Myasnikov Daniil cc7551cf1b test(e2e): use $name in kafka.bats kubectl wait (review feedback PR #2554) 
Address lexfrei's nit on PR #2554: the pre-existing
`kubectl wait kafkas -n tenant-test test ...` on line 47 hardcoded the
resource name, while the existence backstop one line above was already
parametrized with `$name` in 12ccc53d. Switch the wait to `$name` for
consistency so changing the test's resource name in one place keeps the
two lines aligned.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-05 22:10:16 +05:00
Myasnikov Daniil fe138afab7 test(e2e): add --ignore-not-found to vminstance.bats end-of-test deletes 
Symmetrize the post-success cleanup on lines 83-84 with the pre-clean
shape on lines 12-13/41. Without --ignore-not-found, a race that already
removed the resource (e.g., a parallel teardown or finalizer-driven GC)
turns a passing test into a failure after assertions have already run.

Review feedback PR #2555 (lexfrei follow-up #1).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-05 22:06:17 +05:00
Myasnikov DaniilandClaude Opus 4.7 25180c58f7 fix(virtual-machine): bump HR readiness wait from 5s to 60s 
5s was too tight under runner load — both vm-disk-$name and
vm-instance-$name HR waits could miss the operator's reconcile
window. Aligns with the other timeout bumps in this PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-05 13:40:57 +05:00
Myasnikov Daniil 12ccc53d9d test(e2e): tighten event-driven backstops 
- kafka.bats: use $name in the kafkas existence backstop, matching the
  variable used in the kubectl wait below it (consistency with the rest
  of this PR's backstops).
- e2e-install-cozystack.bats: the Tenant→HelmRelease materialisation is
  asynchronous (cozystack-controller observes the Tenant CR and creates
  the HR), so guard hr/tenant-test with the same existence backstop
  pattern used elsewhere.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-05 12:58:58 +05:00
Myasnikov Daniil 9a71dc1427 test(e2e): replace fixed timeouts with event-driven backstops 
Sweep across `hack/*.bats` and `hack/*.sh` replacing fragile
`sleep N` / `timeout N until …; do sleep …; done` patterns with
`kubectl wait --for=condition=…` plus a small existence backstop:

  until kubectl get <resource> >/dev/null 2>&1; do sleep 2; done
  kubectl wait <resource> --for=condition=Ready --timeout=…

Existence backstop added to 33 `kubectl wait` sites across 12 bats
files (redis, kafka, openbao, harbor, mongodb, qdrant, vminstance,
clickhouse, mariadb, postgres, bucket, install). Without it,
`kubectl wait` against a not-yet-created resource sits silently for
the full timeout; with it, the test fails fast with a clear
"resource never appeared" error.

OpenAPI tests' fixed `sleep 5` after starting `kubectl proxy` is now
a port-readiness wait. The 3 remaining genuine sleeps (LINSTOR node
membership propagation, MetalLB end-to-end HTTP path, HR-count
emission heuristic) are annotated with TODO + rationale so a future
reader knows they are intentional.

Same green-path semantics; failures now surface in seconds instead
of after a 5-minute timeout.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
 2026-05-02 18:00:03 +05:00
Myasnikov Daniil 4a5946a8a0 fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts 
Three fixes for the vminstance bats, surfaced once the 3x retry was
dropped from `Run E2E tests`:

1. Disk delete-recreate race. The teardown used
   `kubectl delete vmdisk … || true` and immediately created a new
   disk; sometimes the new disk reconciled before the old finalizer
   completed, leaving a duplicate PVC and the new VM stuck. Drop the
   `|| true`, block until the old disk is gone, then create.

2. VM IP wait timeout 20s -> 120s. Under nested-virt + concurrent
   load, a fresh VM sometimes takes 60-90s to acquire its IP via
   DHCP. The 20s bound was tripping deterministically.

3. VM ready timeout 20s -> 60s. Same root cause; the readiness
   probe needs the IP to be reachable, so the lower bound has to
   grow alongside the IP wait.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
 2026-05-02 17:58:26 +05:00
Andrei Kvapil b5a1ae57de feat(etcd): update operator to v0.4.3 and expose S3 backup schedules (#2428) 
## What this PR does

Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.

Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.

On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).

Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.

`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.

Depends on #2427.

### Scope deferred to follow-ups

- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.

### Release note

```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.

* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.

* **Documentation**
  * Backup configuration guide and parameter reference added.

* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-01 13:16:08 +02:00
Aleksei Sviridkin dff5f8d19a feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes (#2454) 
## What this PR does

Replace the ephemeral `emptyDisk` volume on tenant Kubernetes worker VMs
with persistent PVC-backed storage via KubeVirt `dataVolumeTemplates`.
Previously, kubelet certificates, kubeconfig, and containerd state were
lost on VM reboot, causing the node to fall out of the tenant cluster.

**Changes:**

- Rename `ephemeralStorage` field to `diskSize` in NodeGroup
configuration
- Replace `emptyDisk` volume with `dataVolumeTemplates` using CDI blank
source
- Add `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work
around CDI volume population issues with LINSTOR CSI
- Add optional per-nodeGroup `storageClass` field (defaults to cluster
default)
- Set fixed 2Gi ephemeral-storage resource for virt-launcher pod
- Rename internal mount paths from `/ephemeral` to `/persistent`
- Add helm unit tests (`cluster_test.yaml`) covering dataVolumeTemplates
defaults, custom disk size, storageClass presence/absence, fixed
ephemeral-storage, disk name consistency, and CDI populator annotation
- Quote storage values in templates (`diskSize`, `storageClassName`) for
Helm correctness
- Guard bind-mount commands with `mountpoint -q` checks and create
target directories to handle fresh images and cloud-init retries
- Set `blockSize.matchVolume` on worker node disks for automatic block
size detection, fixing DRBD 4Kn compatibility with QEMU

**Known gotchas:**

- CDI volume population (prime PVC rebind) causes `ClaimMisbound` errors
with LINSTOR CSI driver, regardless of StorageClass (`replicated` or
`local`). The `usePopulator: false` annotation switches to the legacy
import path which works correctly. This is a known CDI issue documented
in
[kubevirt/containerized-data-importer#3146](https://github.com/kubevirt/containerized-data-importer/issues/3146).
- The `replicated` StorageClass with DRBD `suspend-io` quorum policy can
cause transient VM IO errors during LINSTOR satellite restarts (quorum
loss). The `local` StorageClass is recommended for worker node disks.
When `storageClass` is not specified in a nodeGroup, the cluster default
StorageClass is used.
- Existing clusters will trigger a rolling replacement of all worker
nodes on upgrade due to changed KubevirtMachineTemplate hash.
- DRBD volumes use 4096-byte logical sectors (`block-size 4096`).
Without explicit block size configuration, QEMU defaults to 512-byte
sectors, causing immediate IO errors and VM pause. The
`blockSize.matchVolume` setting in the disk spec ensures KubeVirt
auto-detects the correct sector size from the volume.

**Tested on a live cluster:**

- Verified DataVolume provisioning with `local` StorageClass
- Verified nodes join the tenant cluster and reach Ready state
- Verified node survives VM reboot (pod deletion) and returns to Ready
with the same Node object, reusing kubelet certificates from persistent
disk

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes

Worker node VMs now use persistent PVC-backed disks instead of ephemeral emptyDisk volumes. Kubelet certificates, kubeconfig, and containerd state survive VM reboots, preventing nodes from falling out of the tenant cluster.

BREAKING CHANGE: The `ephemeralStorage` field in nodeGroups has been renamed to `diskSize`. A new optional `storageClass` field allows per-nodeGroup storage class selection. Users must update their values overrides accordingly.

Worker node disks now use `blockSize.matchVolume` to automatically detect the underlying volume's sector size, ensuring compatibility with DRBD-backed storage classes that use 4096-byte logical sectors.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional storageClass parameter to select storage class for worker
node persistent disks (defaults to cluster default).
  * Makefile test target added to run Helm unit tests for the chart.

* **Configuration Changes**
* Worker node storage parameter renamed from ephemeralStorage to
diskSize (default: 20Gi).
* Kubelet/containerd state moved to persistent disk paths and node
ephemeral-storage accounting fixed to 2Gi; disk provisioning aligns
block sizing with backing volumes.

* **Documentation**
* Updated docs and chart/schema to reflect the persistent disk model and
new fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 12:41:34 +03:00
ohotnikov.ivan f8001e23b6 fix(postgres): address review feedback 
- Regenerate deepcopy artifacts to remove orphaned PostgreSQLParameters methods
- Update e2e test to use string value for max_connections parameter
- Fix template to avoid .Values mutation and use quote function for safe YAML escaping

Signed-off-by: ohotnikov.ivan <ohotnikov.ivan@e-queo.net>
 2026-04-30 01:43:42 +03:00
Arsolitt 633ff1a617 chore(kubernetes): merge main to resolve CRD schema conflict 
Incorporate images.waitForKubeconfig support from main while
keeping the persistent-storage diskSize and storageClass fields
introduced by this branch.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-27 16:14:07 +03:00
Aleksei Sviridkin cc7d09b458 fix(tests): use far-future cron in etcd backup e2e to avoid job churn 
Address review feedback from coderabbitai on
hack/e2e-apps/etcd.bats:89: the test verifies that the chart renders
EtcdBackupSchedule and that the etcd-operator materializes a CronJob,
not that backups fire. Use "0 0 1 1 *" so no Jobs run against the
intentionally invalid endpoint during the ~6-minute test window.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-24 13:44:21 +03:00
Aleksei Sviridkin 97eef022e5 fix(tests): poll for etcd HelmRelease existence before waiting on readiness 
Address review feedback from coderabbitai on
hack/e2e-apps/etcd.bats:39-40: replace the sleep-then-wait pattern with
a wait_etcd_hr_ready helper that polls hr/etcd until the Cozystack API
reconciles the Etcd CR and only then waits for condition=ready.
kubectl wait exits immediately when the resource does not yet exist,
which caused spurious failures when HelmRelease creation took longer
than 5s.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-24 13:44:02 +03:00
Aleksei Sviridkin 86a93af283 fix(tests): stop masking etcd e2e cleanup failures 
Address review feedback from coderabbitai on
hack/e2e-apps/etcd.bats:9-14: drop the '|| true' guards in setup() so
real teardown failures (RBAC errors, stuck finalizers, wait timeouts)
fail the test setup instead of letting the next test run against stale
HelmRelease, Secret, or EtcdBackupSchedule state. --ignore-not-found
still covers the empty-namespace case on a clean run.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-24 13:43:23 +03:00
Aleksei Sviridkin f8f3b29480 fix(tests): correct stderr redirection order in etcd diagnostics 
Address review feedback from coderabbitai on
hack/e2e-apps/etcd.bats:19-21: redirect stdout to fd 3 first, then dup
stderr onto the new fd 1 so failures from kubectl (NotFound, RBAC,
timeouts) reach the test diagnostics stream instead of being swallowed
by the bats-captured default stdout.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-24 13:42:37 +03:00
Arsolitt 7b05ce0573 fix(kubernetes): update e2e test to use renamed diskSize field 
The e2e test still used the old ephemeralStorage field name,
which triggers the migration guard fail() in the Helm template.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-24 11:21:41 +03:00
Aleksei Sviridkin 4d7b4bbb33 fix(tests): replace bats run idiom in etcd e2e test 
cozytest.sh does not implement the bats 'run <cmd>' helper and '$status'
variable, so the 'Create Etcd with empty backup block (disabled by
default)' test died with 'run: not found' on every attempt. Replace the
bats idiom with plain shell: use '!' prefix and redirect stderr to
assert the resources absent-when-backup-disabled invariant directly.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-21 11:03:27 +03:00
Aleksei Sviridkin ed77e9c6e3 fix(etcd): add helm unit tests, simplify backup guards, harden e2e 
Fourth review round fixes.

Simplify backup validation by dropping placeholder-regex guards and switching default values of s3AccessKey, s3SecretKey, destinationPath, and endpointURL from angle-bracketed hints to empty strings. A single 'required' check on each field is now the whole validation for presence; the regex guard was brittle (silently stopped matching if the hint ever changed) and dropping it makes the values.yaml defaults match the behavior contract the README describes. s3:// prefix and '//' rejection still live in etcd-backup-schedule.yaml as template-time fail branches.

Add helm-unittest suites in packages/extra/etcd/tests/ covering both templates end-to-end: rendered output assertions for the happy path (bucket/key split, region passthrough, conditional rendering of optional fields) and failedTemplate assertions for every fail branch. A 'test' target is wired into the package Makefile so hack/helm-unit-tests.sh picks it up on 'make unit-tests'.

Add type: Opaque to backup-secret.yaml for consistency with the mongodb template.

E2E: wait for HelmRelease, Secret, and EtcdBackupSchedule deletion in setup() so the empty-backup case does not flake against lingering state from the previous test. Replace the fictitious minio-e2e hostname with no-such-endpoint.invalid and explicitly document that the e2e verifies schedule/CronJob materialization, not backup execution.

Restore via EtcdCluster.spec.bootstrap and the one-shot EtcdBackup CRD are still not exposed through this chart; both are explicit follow-ups called out in the README.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-20 17:01:58 +03:00
Aleksei Sviridkin 12fc9cd415 fix(etcd): address third review round 
- Use name='etcd' in all three e2e cases. The chart pins the Helm
  release name to 'etcd' via check-release-name.yaml, so any other
  name causes template rendering to fail before the HelmRelease
  can reconcile. This is why the tenant chart hardcodes HR name
  'etcd' in packages/apps/tenant/templates/etcd.yaml.

- Consolidate all backup validation guards into
  etcd-backup-schedule.yaml so Helm's alphabetical render order no
  longer determines which failure a user sees first. Previously, a
  bad destinationPath would be masked by the earlier placeholder
  check in backup-secret.yaml.

- Use regex match '^<.*>$' for placeholder detection instead of
  exact string equality. The previous check silently stopped
  firing if anyone updated the default placeholder string in
  values.yaml; the regex form catches any angle-bracketed default.

- Reject destinationPath strings containing '//' after the s3://
  prefix. Previously 's3://bucket//subdir' silently produced a key
  beginning with '/', confusing S3 clients and retention policies.

- Document the backup contract in README.md: what enabling it
  produces, which fields are required, why placeholders are
  rejected, and that restore is a known follow-up gap.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-20 16:37:26 +03:00
Aleksei Sviridkin 561bc4dc23 fix(etcd): address second review round 
- Mark backup.enabled as optional in the values typedef so
  generated JSON schema no longer lists it in 'required'.
  Previously, 'backup: {}' (relying on the enabled:false default)
  failed schema validation at CRD admission.

- Validate backup.destinationPath has s3:// prefix. Without this
  check, passing a path without the scheme silently used the
  first path segment as the bucket name.

- Add tail newline to values.schema.json to match repo convention
  (cozyvalues-gen output lacks it — worth fixing in that project
  separately).

- Add bats setup() that clears tenant-test etcd resources before
  each test so a failure mid-test does not leak into subsequent
  tests. Hardcoded chart-level resource names ('etcd',
  'etcd-s3-creds') make two Etcd apps in the same namespace
  collide; setup() enforces a single owner per test.

- Add dump_diagnostics() helper that dumps operator logs + the
  relevant CRD state on wait-for-ready timeouts, so CI failures
  surface actionable context instead of bare exit code 124.

- Add a third e2e case 'Create Etcd with empty backup block' that
  asserts backup:{} admits and produces no backup resources — a
  regression test for the required:['enabled'] bug fixed here.

Known scope: the upstream EtcdCluster.spec.bootstrap restore field
(also new in v0.4.3) is not exposed through this values.yaml yet.
Restoring a cluster from an existing snapshot is a follow-up PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-20 16:15:56 +03:00
Aleksei Sviridkin 0e03ed9ad4 fix(etcd): reject placeholder backup values and harden e2e 
Review found three issues:

1. The guard in backup-secret.yaml only triggered on empty strings,
   not on the placeholder defaults ('<your-access-key>' etc). A user
   enabling backup.enabled=true without overriding the values would
   produce a Secret containing the literal placeholder, and the
   backup CronJob would fail at runtime with an opaque 403 far from
   the source. Add explicit template guards that fail rendering when
   any of the four required fields still hold their placeholder
   default.

2. The backup e2e test waited on .spec.schedule of the newly-created
   EtcdBackupSchedule, which is set at apply time — the wait was
   tautological and did not prove the etcd-operator reconciled
   anything. Replace it with a wait on the EtcdCluster's Ready
   condition plus a poll for the CronJob the operator generates
   from the schedule (selected via the
   etcd.aenix.io/etcdbackupschedule-name label).

3. The backup e2e test also did not wait for EtcdCluster readiness
   before tearing down. Add the same Ready wait as the baseline
   test, and verify the Secret contains the real access key value
   rather than just existing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-20 16:08:43 +03:00
Aleksei Sviridkin ee8ceecc0e test(etcd): add e2e for etcd application and backup schedule 
Adds two bats cases: a baseline Etcd managed-app deployment and a
deployment with backup.enabled=true that verifies the
EtcdBackupSchedule resource and the S3 credentials Secret are
materialized. The backup case uses fake S3 credentials — the
operator will not successfully push backups, but the schedule
resource itself is reconciled, which is what this test validates.

Also hardcodes the EtcdBackupSchedule and backup Secret names to
'etcd' / 'etcd-s3-creds' to match the existing hardcoded
EtcdCluster name in the chart (one etcd per tenant namespace).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-20 15:58:55 +03:00