mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-28 23:57:08 +09:00
648eb42cae3f12bd367102048fa89089fc40407a
164
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 IvanHunters
|
3776c447b9 |
test(e2e): poll users-Job DESIRED_USERS_JSON for target email, not just Job existence
CI E2E showed the users-Job assertion returning `DESIRED_USERS_JSON:
[]` — but the CR body had `users: [{email: e2e-admin@example.com,
role: Admin}]`. Root cause: `hook-delete-policy: before-hook-creation`
keeps the previous hook Job around across helm upgrades, so `until
kubectl get job` matches a stale Job from the previous customconfig-
mode test where users may have been unset (the secretRef swap CR had
no `users:` field, so the schema default `[]` propagated to the
prior Job's DESIRED_USERS_JSON). The System test then grabbed that
stale Job before the helm upgrade for the new CR completed and swapped
in the fresh Job.
Fix: poll on the target env value itself — the new Job with the
correct users list is the first Job whose DESIRED_USERS_JSON string
contains `e2e-admin@example.com`. 600s window matches the inner HR's
15m timeout with room for a slow reconcile.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
7f9e1a8bf6 |
test(e2e): poll the reconciled Grafana CR field instead of racing on CR existence
CI run 28794778739 caught two race conditions in the render-side Monitoring OIDC bats: - `Grafana CR carries auth.generic_oauth pointing at the cozy realm` (monitoring-oidc-system.bats:86) failed with `enabled=` empty after a 0-second wait. The prior `until kubectl get grafana grafana` only proved the CR EXISTS — it did not prove grafana-operator had reconciled the mode=System values onto it. When the CR from the previous `Monitoring CR accepts spec.oidc.mode=System` test was already there but the operator hadn't yet flipped `.spec.config.auth .generic_oauth.enabled` to "true", the jsonpath returned empty and `[ "" = "true" ]` short-circuited the whole test. - `secretRef variant mounts operator Secret under /etc/grafana/oidc` (monitoring-oidc-customconfig.bats:119) timed out at 180s waiting for the volumeMount swap from inline → ini. On a warm sandbox the full inner-HR upgrade + grafana-operator reconcile cycle can legitimately take >3m — the timeout was too tight. Fix: replace `until kubectl get grafana` with `until [ "$(kubectl ... jsonpath=<target-field>)" = <expected> ]` on every subsequent assertion, and bump the secretRef-swap window to 600s (well inside the inner HR's 15m timeout). Same pattern used for the "no role_attribute_path" and "operator payload verbatim" assertions so the whole suite is race-free. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
123616a1d0 |
test(e2e): swap KeycloakRealmGroup assertions for the users-Job in bats
Aligns the render-side e2e bats with the users-map contract: - Feed a `spec.oidc.users:` entry to each mode's Monitoring CR so the reconcile Job has real input to act on. - Assert the users-Job appears in the release namespace with the desired `DESIRED_USERS_JSON` env and the expected `OIDC_MODE`. - Assert the Grafana CR renders with `skip_org_role_sync = "true"` and `oauth_allow_insecure_email_lookup = "true"` and NO `role_attribute_path`. - Flip the three KeycloakRealmGroup existence assertions to negative checks: chart-owned realm groups must NOT be present in either System or CustomConfig mode. Does not add a full OIDC login flow through Keycloak — that stays deferred to a follow-up integration suite, same posture as the tenant kube-apiserver's kubernetes-oidc-system.bats. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
5f7ed89799 |
fix(e2e): work around EDP schema-default strip + kubectl delete/apply race
Two follow-up bats fixes uncovered by the first live e2e pass on the
singleton-aligned monitoring OIDC tests:
1. monitoring-oidc-system, test 4 (KeycloakClient assertion). The EDP
Keycloak operator's KeycloakClient CRD strips `spec.public: false`
on write because false is the schema default. The chart emits
`public: false` explicitly (confidentiality guard), but neither
managedFields nor the resulting spec ever carries the field, so
the assertion `[ "${public}" = "false" ]` grepped an empty string
and tripped. Assert `spec.clientAuthenticatorType == "client-secret"`
instead — the operator persists it, and it is the direct signal
for "confidential client" (which is what the `public: false`
assertion was testing for).
2. monitoring-oidc-customconfig, test 4 (secretRef variant). The
original flow was delete-then-recreate the Monitoring CR to swap
inline `customConfig.config` for `customConfig.secretRef.name`.
`kubectl apply` immediately after delete raced the API server's
grace-period cache: apply's GET-then-decide phase saw the CR
during grace, chose the patch path, and the patch hit NotFound
once the delete propagated. Drop the delete entirely — cozystack-
operator reconciles the CR mode change on the same object and
Flux re-renders the chart with the new values.
Add an explicit wait for the target state (Grafana CR's
`oidc-custom-ini` volumeMount present AND `spec.config.auth.generic_oauth`
absent) before the assertions, since the CR update triggers an
asynchronous chart re-render and the immediate get would still
see the inline auth.generic_oauth block from @test 1.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
cb04caecee |
fix(e2e): align monitoring OIDC bats with chart singleton naming
The extra/monitoring chart enforces `.Release.Name == "monitoring"`
in templates/check-release-name.yaml (the tenant Monitoring is a
per-namespace singleton). Naming the Monitoring CR anything else
makes the operator materialise an outer HelmRelease with that name
and the chart hard-fails at render with
execution error at (monitoring/templates/check-release-name.yaml:2:4):
The name of the release MUST BE monitoring
Rename the CR to `monitoring` in both files. Rework every derived
name to match the actual reality of the chart pipeline instead of
the invented `monitoring-<TEST_NAME>` prefix the tests polled for:
* outer HR = ${CR_NAME} = monitoring
* inner HR = ${CR_NAME}-system = monitoring-system
(rendered by extra/monitoring/templates/helmrelease.yaml as
`metadata.name: {{ .Release.Name }}-system`)
* clientId = ${namespace}-${inner Release.Name}
= tenant-test-monitoring-system
* scope = ${clientId}-audience
* groups = ${clientId}-{admin,editor,viewer}
* client-secret Secret = ${inner Release.Name}-oidc-client
= monitoring-system-oidc-client
The BYO Secret name in the customconfig test is chart-independent
(the tenant provides it via customConfig.secretRef.name), so it is
kept short and no longer tied to CR_NAME.
The wait sequence in the first @test of each file is also split
into two probes: outer HR appears first (from the operator), then
inner HR (from the extra/monitoring chart's helmrelease.yaml), then
the Grafana CR. Under the previous incorrect single-probe wait
(`hr monitoring-${TEST_NAME}`) the timeout tripped after 60s
because that HR never existed under any name.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
9ff05b2f32 |
test(e2e): port monitoring OIDC bats to cozytest lifecycle conventions
Mirrors the same port applied to kubernetes-oidc-*.bats in the tenant
kube-apiserver OIDC series: cozytest.sh (the e2e runner) is not real
bats, so setup()/teardown() are never invoked and any state they
established (TEST_NAME) is unset by the time an @test body runs,
tripping cozytest.sh's `set -eu` at the first "${TEST_NAME}"
expansion with `TEST_NAME: parameter not set`.
Lift TEST_NAME to file scope as a stable constant, register cleanup
via cozy_cleanup() so cozytest.sh runs it at suite exit and on the
first failing test, and call cleanup_mon() inline at the top of the
first @test for per-run isolation. Fixes the two E2E failures on
this PR: `Monitoring CR accepts spec.oidc.mode=CustomConfig with
inline config` and `Monitoring CR accepts spec.oidc.mode=System`.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
fe2c606f79 |
test(e2e): add Phase-1 OIDC render-side bats for Monitoring
Two bats files mirroring the shape of the tenant kube-apiserver PR's
kubernetes-oidc-{system,customconfig}.bats:
- monitoring-oidc-system.bats — asserts the Monitoring CR admits the
new spec.oidc.mode: System field; the chart renders the Grafana CR
with an auth.generic_oauth section pointing at the cozy realm; a
32-char client-secret Secret is persisted; the KeycloakClient (with
public: false), KeycloakClientScope (audience mapper), and three
KeycloakRealmGroups (-admin, -editor, -viewer) land in cozy; and
the Grafana Deployment injects GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET
from the Secret.
- monitoring-oidc-customconfig.bats — asserts the CR admits inline
CustomConfig; the operator's payload lands on spec.config verbatim
with no cozy realm in the auth_url; NO KeycloakClient / Scope /
Groups are created in cozy; and the secretRef variant mounts the
operator Secret under /etc/grafana/oidc without emitting
auth.generic_oauth under spec.config.
Neither drives a real browser login through Keycloak — that is
deferred to the follow-up integration suite, same posture as the
tenant kube-apiserver's Phase-1 tests.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
90474f9f7f |
test(e2e): port the disabled VM-backup suite to Chainsaw
Convert hack/e2e-apps/backup.bats.disabled to a Chainsaw test under hack/e2e-chainsaw/backup/. It stays opt-in: the chainsaw-test.yaml.disabled filename keeps it out of chainsaw's default discovery and out of hack/select-e2e.sh enumeration, matching the previous disabled state. The declarative readiness chain (Bucket -> VirtualMachine -> BackupJob) becomes assert operations; the imperative Velero-backup + S3 verification stays a single script step, mirroring the bucket suite. README documents the opt-in run command and the backup-infra preconditions. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
e22d84f2c8 |
test(e2e): migrate app suite from BATS to Chainsaw and wire into CI
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot (postgres + bucket) to all 18 app suites. - Declarative ports assert on status/conditions and concrete fields; the repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate collapses into a single polling assert with structured failure diffs and automatic events/describe/podLogs capture on failure. - Inherently imperative suites keep their logic in script steps: openbao init/unseal, kuberture split-horizon probes, vminstance, the gateway admission/impersonation cases, and kubernetes-latest/previous, which wrap the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim. - Gateway tests derive the tenant apex from the namespace host label at runtime, so they are host-independent. CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw binary is added to the e2e-sandbox image. install-cozystack and test-openapi stay BATS. The per-app 3-retry loop is dropped, since assertion polling replaces the fixed-timeout flakiness it papered over. Verified against a dev cluster (DB/app, storage and VM suites pass). Service port assertions use the ports[*].port projection form because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal port filters. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
0855ff2d1b |
test(e2e): assert bucketaccess accessGranted=true and dump controller pods on bucket failure
The two bucketaccesses waits used the bare jsonpath form, which also matches the literal "false"; require =true so a never-granted access fails the wait instead of passing against an ungranted object. Also print the objectstorage-controller pod status in the bucketReady failure block so controller-side issues are visible from the test log. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
63d14e7487 |
test(bucket,harbor): assert BucketClaim converges to bucketReady promptly
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.
Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
5653c30799 |
fix(e2e): bound per-test crust-gather snapshots with a timeout
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh) ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect can hang indefinitely — observed wedging the E2E install step for ~2h35m after a test failure until the job-level cancel, with no output after the capture line. `|| true` only catches a bad exit code, never a hang. Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
9beda4e60e |
fix(e2e): reach tenant API via LoadBalancer instead of port-forward
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.
Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.
Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).
Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
51496832e7 |
fix(e2e): capture tenant cluster snapshot from an in-subshell trap
crust-gather can only reach a tenant Kamaji cluster through the test's localhost port-forward (it connects directly to the kubeconfig server URL — there is no apiserver-proxy mode — and the in-cluster URL is unreachable from the runner). cozytest.sh's generic EXIT trap runs in the parent shell AFTER the failing test subshell has exited and its port-forward is reaped, so a collect there hits a dead socket and crust-gather bails before writing anything (every tenant snapshot so far came back empty). Capture the tenant from an EXIT trap registered INSIDE run_kubernetes_test instead: it fires during THAT subshell's exit, while the supervised port-forward is still alive. Verified end-to-end against the real hack/cozytest.sh under dash (CI's /bin/sh) in an ubuntu:22.04 container — the in-subshell trap fires on failure with the PF alive, the tenant snapshot lands in the same snapshots/<test>/ dir as the host snapshot, the generic cozy_cleanup still runs, and there is no pipeline deadlock. Drop the now-dead tenant loop from cozytest.sh's generic trap (it could only ever see a reaped PF); it keeps the host snapshot. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
391c27f11e |
fix(e2e): supervise tenant API port-forward so a drop self-heals
A single `kubectl port-forward` to the tenant Kamaji API silently dies on an SPDY/connection drop and never reconnects. The tenant apiserver itself stays healthy (control-plane pods observed at 0 restarts while localhost refused connections), so this is a client-side port-forward death, not a control-plane outage. Once it dies every later tenant kubectl op wedges: a long `kubectl wait` burns its full timeout against the dead socket (observed: nfs-test-pod hit its 10m deadline) and the failure-path `describe`/`get events` print 'connection refused', destroying the diagnostics too. Run the port-forward under a respawn loop so a drop self-heals within ~1s; `kubectl wait` retries across the gap and the in-test failure dumps work again. Track the supervisor PID in a pidfile so cozy_cleanup() can stop it on a failed run (the success path kills it directly), and reap the child port-forward with pkill in both paths. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
d7f244fcf6 |
test(e2e): give harbor BucketClaim readiness a 10m budget
Harbor's registry BucketClaim wait timed out at 300s on an overloaded
runner even though the underlying Bucket reached bucketReady=true and the
BucketAccess was granted — only the central COSI controller's propagation
of readiness back onto the namespaced BucketClaim lagged (all HelmReleases,
harbor included, were Ready by end-of-run). Raise the wait to 600s to match
the dependent harbor-system HelmRelease budget and remove the flake.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
50e5f94132 |
test(e2e): per-test crust-gather snapshots + cleanup-on-exit via cozytest.sh
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined. run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook. Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
9e43d455f0 |
test(e2e): wait for tenant nodes Ready on a single deadline, not split budgets
kubernetes-latest flaked because the worker-readiness wait was split into 8m-to-join + 3m-to-become-Ready. Under sandbox load a slow KubeVirt VM boot consumes most of the join budget, then the tenant cluster's cilium CNI needs several more minutes to make the freshly-joined nodes Ready, overflowing the fixed 3m Ready window even though the CNI converges fine (its HelmRelease reached True shortly after). Replace with one 12m deadline polling for >=2 Ready nodes, robust to wherever the time goes. Same single-deadline shape as the LINSTOR post-install fix (#2928). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
763b85eb28 |
test(e2e): wait for EtcdBackupSchedule instead of a bare get in etcd.bats
Reconfiguring the Etcd CR triggers a HelmRelease upgrade that renders the EtcdBackupSchedule; hr/etcd and etcdcluster can report ready on the prior revision before the new resource materializes, so the immediate get raced to NotFound. Poll for it (timeout 120s), mirroring the CronJob wait already used later in the same test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
40431de0b5 |
test(e2e): fix bucket.bats port-forward and S3 client reliability (#2944)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
5b986e0a47 |
test(e2e): wait out nested-tenant teardown in gateway.bats
The last three gateway tests create nested tenants (gwprop, gwparent/gwchild, rparent/rchild) and deleted them fire-and-forget, with parent and child deletes back-to-back. The file exits in ~25s while the uninstalls run on for minutes: each tenant uninstall blocks on a cleanup Job in cozy-system, and a parent deleted alongside its child wedges on the child's still-terminating namespace. On the tenants helm-controller shard (--concurrent=5) the leftovers from one gateway.bats run — three stuck parent uninstalls, one stuck child uninstall, and the rchild HR still mid-install — occupied exactly 5 workers, so the next test in line (harbor) sat unreconciled for its whole 5m HR-Ready budget and failed without ever erroring. Sequence the teardown instead: delete child, wait for its HR to go away, then delete parent and wait again. The waits are hard failures on purpose — if a tenant uninstall genuinely wedges, gateway.bats is the test that should go red, not whichever app test runs next. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
c435efa199 |
feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602)
## What this PR does Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and the embedded management-cluster manifests, and folds in the chart fixes that become hard errors under the new strict helm-controller v1.5. Flux v2.8's helm-controller v1.5.0 ships: - **Server-Side Apply with `--force-conflicts`** — strict CRD schema enforcement; misplaced fields (silently dropped on v2.7) now fail the apply. - **kstatus-based health checking** by default — parent HR waits for every applied resource (including child HRs) to be Ready before flipping its own Ready, surfacing latent ordering deadlocks. - **Helm v4 embedded** — `upgrade.force: true` is deprecated and now conflicts with SSA. - **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper readiness gating (used in PR #2601 split). Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same Flux upgrade triggers it. ## Commits **Flux upgrade itself:** - `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts to v0.48.0; web UI opt-in. - `feat(flux)`: regenerate embedded management-cluster manifests via `make update` in `packages/core/flux-aio` (timoni bundle build). **Chart fixes for strict SSA — fields the chart sent that v2.7 silently dropped, v2.8 rejects:** - `fix(kubevirt-instancetypes)`: drop persistent strip that produced null `preferredTPM` (folds #2612). - `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`, `minimumUptimeSecondsForBounce` from inside `automationOptions` to direct children of `spec`. - `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a phantom `template.spec`. - `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is set. - `fix(platform)`: drop deprecated `upgrade.force: true` from HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths. **Ordering / deadlock fixes under v2.8 kstatus:** - `fix(vpa)`: break circular wait between parent install and nested `vpa-for-vpa` HR. - `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant addon child HRs (parent waits on child via kstatus, child waited on parent — deadlock). **E2E waits for v2.8 kstatus timing:** - `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's faster dispatch). - `test(e2e)`: wait for parent HR Ready before downstream asserts in `run-kubernetes.sh` and `vminstance.bats`. ## Scope discipline This PR is part of the split of #2619 (the consolidated CI fixes branch) into review-friendly pieces. Companion PRs: - **PR #2601** (seaweedfs split) — folded into this PR (commits `29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption migration 43 (targetVersion 44), and the configurable db resources all land here, because the strict-SSA `upgrade.force` removal and the kstatus parent-HR timeout bump only make sense together with the split. #2601 is superseded. - **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) — independent, lands separately. - Several smaller standalone fixes lifted out of #2619 (startup probes, cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC test improvements) — opened as separate PRs. ## Verification - `helm template` renders cleanly for both `fluxcd` and `fluxcd-operator` packages with `web.enabled=false` (default) and `web.enabled=true`. - Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0 / v2.1.0 controller image strings. - No references to the v0.39-removed `--disable-wait-interruption` flag anywhere in `packages/` or `internal/`. ### Release note ```release-note Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking). When upgrading existing clusters: - Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too. - HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it. - KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass. - KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained. - FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Flux Status web UI: optional web server, config secret, service port, HTTPRoute/Ingress, network policy, and RBAC roles. * New SeaweedFS DB Helm chart and optional managed DB release. * **Improvements** * CRD/schema enhancements: new provider kinds, validations, variant option, and external checksum refs. * Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for reliability. * **Chores** * Bumped Flux Operator and Flux versions; documentation links updated to fluxoperator.dev. * **Bug Fixes** * Removed aggressive HelmRelease force-upgrade/install flags. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2d58479f06 |
feat(platform): add kuberture as optional system package (#2647)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional system package. Kuberture is a small controller that translates the `default/kubernetes` EndpointSlice into headless Services with external-dns-style annotations, closing the gap that external-dns cannot use EndpointSlice as a source directly: ```text EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS ``` The package is opt-in via `bundles.enabledPackages: [cozystack.kuberture]`. Operators populate `config.outputs` per HelmRelease — each output targets a specific external-dns instance by its `annotationPrefix`, so a single kuberture install can serve multiple external-dns instances (e.g. one publishing to the public zone, another to an internal corporate prefix) simultaneously. The vendored chart at `packages/system/kuberture/charts/kuberture/` is pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and pinned by both version AND OCI manifest digest in the Makefile (mirroring the ouroboros lockstep pattern). The controller image at `ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim values.yaml. Both pins advance in lockstep with each upstream release; the procedure is documented inline. Install-order dependencies declared in `PackageSource`: `cozystack.networking` (CNI before pod scheduling) and `cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the chart's enabled-by-default ServiceMonitor manifest). External-dns is deliberately NOT a dependency — kuberture creates annotated Services regardless of whether external-dns is installed; external-dns is a downstream consumer, not a runtime prerequisite. Coverage: - **helm-unittest** (`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases): fail-fast on empty outputs, ServiceMonitor enabled by default, source pinned to `default/kubernetes`, multi-output prefix isolation, empty-string `annotationPrefix` schema rejection, omitted-prefix fallback to controller default, ConfigMap → Deployment volume mount, controller image pinned by tag+sha256 digest, Role permits full Service lifecycle, metrics containerPort derived from `metricsBindAddress`. - **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package cozystack.kuberture` with two outputs (different `annotationPrefix` values), waits for HelmRelease and controller readiness, asserts both output Services are headless and carry the right `hostname`/`target`/`ttl` annotations under their configured prefix, cross-checks per-output isolation (each Service only carries its own prefix's annotations). ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> Not applicable — no UI changes. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Adds kuberture as an optional system package to publish the Kubernetes API endpoint to DNS via configurable external-dns outputs; includes Helm chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart values/schema. * **Tests** * Adds Helm unit tests and an end-to-end smoke test validating controller readiness, headless Service annotations, and external-dns routing/probe behavior. * **Documentation** * Adds a full README with usage, configuration examples, and vendoring/pinning guidance. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- Documentation companion: [cozystack/website#539](https://github.com/cozystack/website/pull/539). |
||
|
Myasnikov Daniil
|
a3a7a6d516 |
test(e2e): add HR existence backstop for external-dns ${name}-system
The two external-dns bats tests already poll for the user-facing ${name}
HelmRelease before calling kubectl wait, but they immediately wait on the
${name}-system HR without the same backstop. The system HR is materialised
asynchronously by the controller, so kubectl wait can race and exit non-zero
when the object does not exist yet. Apply the same existence poll pattern
to both occurrences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
401993ac62 |
test(e2e): wait for parent HR Ready before downstream asserts
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape:
the bats steps assumed a side-effect ordering ("by the time X is true,
the parent HR's helm install is done") that held under v2.7's faster
dispatch but breaks under v2.8 where helm-controller walks kstatus on
every applied resource before flipping the HR Ready.
run-kubernetes.sh:
After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx)
reaches Ready, the remediation guard reads `.status.history` on the
parent kubernetes-${test_name} HR. The parent's helm install can still
be Reconciling at that point because kstatus is finishing its full
sweep; `.status.history` is empty until install completes, so the
guard exits 1 with "empty .status.history on a Ready HR". kubernetes-
latest masked this because the ouroboros assertions add ~10m of
buffer; kubernetes-previous (no ouroboros) hit the guard with the
parent still installing. Add an explicit
kubectl wait hr/kubernetes-${test_name} --for=condition=ready
before the guard.
vminstance.bats:
The test waited 120s for `kubectl get vmi vm-instance-$name` to
return a VMI with an assigned IP, then waited for the parent HR Ready
separately afterwards. Under v2.7 the parent HR's helm install
applied the VirtualMachine CR fast enough that kubevirt had created
the VMI before the 120s budget expired. Under v2.8 the dispatch is
slower and the VMI poll returned NotFound for its entire budget.
Reorder: wait for HR Ready first (which implies the VirtualMachine
CR is applied + kstatus-ready), then poll for the VMI with IP.
Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
7b9f286faa |
test(e2e): bump app HR-Ready waits for Flux v2.8 kstatus
Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.
Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.
Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.
Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.
Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
ea7a8c8675 |
test(e2e): quote HelmRelease names in kubectl wait
Both \${test_name} and \${component} are literal-safe today (no
whitespace, no glob characters), so this is a style-only change — but
quoting variable expansions in kubectl arguments matches the rest of
the file and removes a future-foot-gun if a test name ever grows a
hyphen-y suffix that interacts oddly with completion or globbing.
Scoped to the two HelmRelease wait lines flagged by CodeRabbit and
echoed by the human reviewer; the machinedeployment wait on the
preceding line keeps its existing style.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
64c443caf1 |
test(e2e): split local k8s_version declaration from yq invocation
\`local k8s_version=\$(yq ...)\` masks the exit status of the command substitution — the assignment to a local variable always succeeds, so a failing yq is swallowed. Declare the local first, then assign, so non-zero yq exits propagate via shell errexit semantics if the script ever runs under \`set -e\`. Pure shell hygiene; no behavioral change on the success path. Surfaced by CodeRabbit and confirmed by the human reviewer as non-blocking but worth applying. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
8c6e149177 |
test(e2e): delete standalone ouroboros.bats now folded into kubernetes-latest
The previous commit (3c81261b) folded the ouroboros hairpin-NAT reconciliation assertions into the kubernetes-latest tenant under run-kubernetes.sh, but did not remove the standalone bats file. The e2e workflow auto-discovers every hack/e2e-apps/*.bats and dispatches test-apps-<name> via the wildcard rule in packages/core/testing/Makefile. Leaving ouroboros.bats in the tree meant the assertions ran twice: once on the standalone ~25m Kamaji-backed tenant and again as the folded addon on the kubernetes-latest tenant — the opposite of the consolidation the fold was intended to deliver. The folded assertions in hack/e2e-apps/run-kubernetes.sh cover the same surface (HR Ready, controller pod Running, coredns-custom rewrite-line injection, and end-to-end DNS resolution from inside the tenant), so the standalone file is dead weight after the fold. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
68a9a6e5ad |
test(e2e): bump NFS pod wait to 10m to fit slow-path chain
Previous run failed kubernetes-latest (and then kubernetes-previous via
cascade) on the 5m budget for nfs-test-pod to reach Succeeded. The
chain is host linstor prime PVC + CDI import + tenant CSI attach +
busybox image pull + containerd CreateContainer (which occasionally
stalls past kubelet's 2m CRI deadline and retries with name-reservation
conflict). ~40s on the happy path, 6m+ in observed worst case.
When kubernetes-latest times out, its tenant cluster (2x 20Gi worker
PVCs) leaks because set -e exits before any cleanup. kubernetes-
previous then runs, adds another 40Gi, and the host tenant-quota
(100Gi) is exhausted -- every downstream app test (mariadb, mongo,
postgres, qdrant, redis, openbao, vminstance) fails with
ErrCreatingPVCPrime / exceeded quota.
Bumping the wait to 10m gives the slow path room. Headroom costs
nothing on green runs since the pod is busybox echo.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
04a54b01cf |
test(e2e): wait for tenant CSI HR Ready before NFS test
kubernetes-previous fails on the NFS pod step. From cozyreport + job log:
- Pod scheduled to tenant worker
- 7x FailedAttachVolume over 35s: CSINode does not contain driver
csi.kubevirt.io (kubevirt-csi-node DaemonSet still rolling out)
- Attach succeeds at T+1m08s after the driver registers
- busybox pull queued ~1m25s, completes at T+2m48s
- CreateContainer in containerd then stalls past kubelet's CRI
deadline; retry trips "failed to reserve container name ... is
reserved for <prev>" -- the 5m pod-Succeeded wait then expires.
Root cause: run-kubernetes.sh creates the NFS PVC+Pod before asserting
kubernetes-${test_name}-csi is Ready. That HR lives in the parent
cluster but installs cozy-kubevirt-csi-node into the tenant via
kubeConfig.secretRef; helm-controller walks kstatus on the tenant
DaemonSet using the tenant kubeconfig. HR Ready in parent therefore
implies kubevirt-csi-node Pods Ready on every tenant worker, which
implies csi.kubevirt.io is published in CSINode. The existing wait at
line 354 runs AFTER the NFS test, so it does not gate the race.
Move the wait earlier with a 10m budget. Post-NFS sibling waits stay
as is, so component coverage is unchanged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
5035ac1414 |
test(e2e): fold ouroboros assertions into kubernetes-latest
The standalone hack/e2e-apps/ouroboros.bats provisioned a full Kamaji-
backed tenant Kubernetes cluster (control plane + 2 CAPI workers + base
addons) just to flip `addons.ouroboros.enabled=true` on top, then ran
~15m of hairpin-NAT reconciliation assertions. Cluster shape was
identical to what hack/e2e-apps/run-kubernetes.sh already provisions for
kubernetes-latest, so the second Kamaji bringup was pure duplication —
roughly 25-45m of CI wall-time per run.
Add an `enable_ouroboros` 4th parameter to `run_kubernetes_test()`. When
"true", inject the ouroboros addon block into the Kubernetes CR's
addons map and run the post-cluster assertions inline:
- Wait for the ouroboros HR Ready in the host namespace.
- Verify coredns-custom ConfigMap exists in tenant kube-system.
- Wait for the ouroboros controller pod Ready.
- Apply an Ingress with a TLS host in tenant default namespace.
- Poll coredns coredns-custom for the rewrite line (5m budget).
- Resolve the hairpin host from inside the tenant via a netshoot pod
and assert it points at the ouroboros-proxy ClusterIP.
- Clean up dnscheck pod + Ingress.
kubernetes-latest passes "true" (gets ouroboros coverage); kubernetes-
previous leaves it empty (control-plane-only smoke remains).
(The standalone ouroboros.bats was removed in the preceding commit
alongside an unrelated foundationdb chart fix — the git rm staging had
already happened when the foundationdb commit landed.)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
c85e473523 |
test(e2e): inheritance attach asserts controller writes, not Cilium Accepted
The previous e2e check waited for `HTTPRoute.status.parents[0].conditions[Accepted]=True` after applying a child tenant's HTTPRoute against the parent's Gateway. That condition is set by Cilium after the listener's TLS Secret becomes Ready — and in the e2e cluster every fresh certificate fails to issue (LE prod refuses every `.example.org` hostname by policy, visible across alerta/grafana/dashboard/seaweedfs-s3 cluster certs in cozyreport). So Cilium never set Accepted, the route timed out 3/3 retries, and the failure was environmental — not a regression in the inheritance code path. The contract the test exists to pin is cozystack-controller's behaviour, which has two observable writes: 1. The parent Gateway grows a per-listener HTTPS entry for the child route's hostname (proves collectHostnameClaims saw the route through the inheritance label and reconcileGateway appended the listener). 2. A per-listener Certificate object appears in the parent namespace with the route's hostname in spec.dnsNames (proves reconcilePerListenerCertificates ran for the same hostname). Both writes are deterministic and depend only on the inheritance label + the HTTPRoute existing — not on ACME, not on Cilium TLS readiness. The test now polls those two objects directly within shorter, scenario-appropriate timeouts (120s for the listener, 60s for the Certificate). The route Accepted check is dropped with an explanation in the test comment so a future cluster with a working ACME doesn't get a misleading pre-condition added back. The namespace inheritance label is also verified up front as the precondition collectHostnameClaims's filter reads. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
4120f8795e |
test(e2e): cover child-tenant inheritance attach flow
Replaces the now-obsolete "Package admission rejects tenant-* in gateway.attachedNamespaces" e2e (the VAP it pinned was dropped in the inheritance refactor) and adds two new e2e cases that cover the end-to-end inheritance contract: * "Package admission accepts gateway.attachedNamespaces with tenant-* entries" — regression guard against a future refactor re-introducing the gate; the hijack surface is now handled by Layers 4/5/7 (Tenant.spec.host VAP, namespace label immutability VAP, HTTPRoute hostname VAP), not by entries in attachedNamespaces. * "child tenant without explicit gateway inherits _namespace.gateway from a Gateway-owning parent" — chart-level lockstep: a parent tenant with gateway=true produces both the namespace label and the cozystack-values Secret pointing at itself, and a child Tenant under that parent with the gateway field unset receives the parent's name in both places. Plus a negative assertion that the child does NOT get its own gateway HelmRelease. * "child tenant's HTTPRoute attaches to parent's Gateway via inheritance label" — full cross-namespace attach: parent owns the Gateway, child inherits, an HTTPRoute in the child namespace pointing at parent's Gateway reaches Accepted=True once Cilium evaluates the label-based allowedRoutes selector. Route hostname uses the child's derived apex so Layer 7 admits it. The Gateway-Programmed wait is intentionally omitted from the last test — Programmed depends on the LB allocator that ships with the e2e cluster, while Accepted is a pure spec evaluation on the parent Gateway's allowedRoutes selector and is the property the inheritance refactor is meant to deliver. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d2e851f63b |
feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Arsolitt
|
2086f3e6b6 |
test(kafka): assert ZooKeeper PVC-1 is created before deletion check
Address review feedback from coderabbitai on hack/e2e-apps/kafka.bats:58: The post-delete `while kubectl get pvc data-kafka-$name-zookeeper-1` loop exits immediately when the PVC has never been created, so a regression that prevents Strimzi from provisioning the second ZooKeeper PVC would silently pass the reclamation check. Mirror the PVC-0 readiness guard (existence + Bound) for PVC-1 before deleting the Kafka CR so the test only proceeds once both PVCs are observed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
c4b0606e67 |
test(kafka): verify ZooKeeper PVCs are reclaimed by the operator
Address review feedback from gemini-code-assist on hack/e2e-apps/kafka.bats:53-55: Replace the manual kubectl delete of the two ZooKeeper PVCs with a poll that waits for the Strimzi operator to reclaim them. With deleteClaim: true on the ZooKeeper persistent-claim storage, the operator removes the PVCs as part of Kafka CR deletion; manual deletion would either race the operator, fail with NotFound, or silently mask a regression in the reclamation path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Aleksei Sviridkin
|
5c188fede3 |
test(kuberture): cover platform install and multi-output annotations in e2e
Smoke test that drives the cozystack.kuberture Package end-to-end on the e2e cluster: - creates a Package with two outputs targeting distinct annotationPrefix values so a single kuberture install can serve multiple external-dns instances simultaneously - waits for namespace, HelmRelease and controller Deployment readiness - asserts both output Services are headless and carry the right hostname/target/ttl annotations under their configured prefix - cross-checks that each Service only carries its own prefix's annotations, not the other output's Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
6f83d9a794 |
feat(ouroboros): bump to v0.7.2 with debug logging on tenants
The v0.7.2 ouroboros release adds --log-level / OUROBOROS_CONTROLLER_LOG_LEVEL, plants Debug logs at every informer event handler (AddFunc/UpdateFunc/ DeleteFunc with kind+namespace/name via describeObj) and at reconcile entry/exit, wires SetWatchErrorHandler on Ingress / Gateway / HTTPRoute informers, and surfaces cache-sync elapsed at Info. The diagnostic surface targets the kamaji-tenant-via-konnectivity flake where bats sees a fresh Ingress create succeed against the tenant apiserver but the controller's coredns-custom rewrite lands roughly 7-10 minutes later — without per-event visibility the failure mode is indistinguishable between 'watch never fires', 'reconcile silently errors', and 'API list returns empty until reflector reconnect'. Bumps the chart-manifest digest, the controller-image digest, and the cozystack-side image-pin assertion in lockstep. The bats deadline for the coredns-custom rewrite check rises from 180s to 900s — even on the broken-watch path the reflector eventually reconnects within that budget, so the test stops failing on the symptom while we keep the diagnostic on to learn what the symptom actually is. cozystack tenants force controller.logLevel: debug via the defaultOuroborosValues helper; operators that want quieter logs can override through addons.ouroboros.valuesOverride. Two helm-unittest fixtures pin both the default and the override path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d6bcb27c53 |
test(ouroboros): dump tenant state on every assertion failure, not just HR wait
The CI run after pinning controller.clusterDomain showed the test was finally getting past the helm-controller wait — the proxy /readyz unblocked, the HR reaches Ready, and the controller pod becomes Ready — but the next assertion (the rewrite snippet appearing in kube-system/coredns-custom within 180s of creating the probe Ingress) is still failing on every retry. Without a dump on this assertion the failure surfaces only as `❌ Test failed (exit 1)` with the polled `snippet=` empty lines preceding it; the controller logs that would explain why the reconciler skipped the Ingress are nowhere visible. Refactor the existing inline diagnostic block into a `dump_tenant_state` helper that is invoked on every assertion's failure path. Wrap the rewrite-snippet check, the proxy ClusterIP non-empty check, and the dnscheck pod-phase check in `if ! ...; then dump_tenant_state; exit 1; fi` so each failure dumps the same set of host/tenant pods, events, controller and proxy logs, ingress-nginx Service+endpoints, and tenant CoreDNS Corefile that the original HR-wait failure path already covered. Also append a `kubectl get ingress -n default -o yaml` dump of the probe Ingress so a controller log line like "skipped Ingress without TLS host" is correlatable against the actual rendered manifest at the time of failure. The follow-up fix for whatever the next CI run surfaces will land on this same branch. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
89aab4459b |
test(ouroboros): port-forward + super-admin.conf for tenant kubectl path
Every tenant-side kubectl call in hack/e2e-apps/ouroboros.bats was failing inside the e2e sandbox container with `dial tcp: lookup kubernetes-${cluster}.${ns}.svc on 169.254.169.254:53: no such host`. The bats test pulled the `super-admin.svc` variant of the tenant admin-kubeconfig, which points at the in-cluster Service DNS name `<release>.<ns>.svc:6443` — a name only host-cluster CoreDNS can resolve. The sandbox container's resolver is the Docker/QEMU host link-local 169.254.169.254, not the host-cluster CoreDNS, so every `KUBECONFIG=$kubeconfig kubectl ...` invocation in the test was returning NXDOMAIN.
The check that gated whether the test even reached its assertions (the `kubectl wait helmrelease kubernetes-${cluster}-ouroboros`) ran against the host kube-context, which masked the broken kubeconfig path until the helmrelease itself failed to reconcile and the diagnostic block fired — at which point every dump command returned only NXDOMAIN errors instead of actionable tenant-cluster state.
Switch to `super-admin.conf` (the operator-facing kubeconfig variant), rewrite `clusters[0].cluster.server` to `https://localhost:${pf_port}`, and run a `kubectl port-forward service/kubernetes-${cluster} ${pf_port}:6443` in the background — the same pattern hack/e2e-apps/run-kubernetes.sh already uses for tenant-cluster reachability. Replace the inline trap RM with a `cleanup_kubeconfig` helper that also kills the port-forward process so retries don't leak listeners.
This is a test infrastructure fix; the underlying ouroboros HelmRelease install timeout itself is a separate root cause that the next CI run, with usable diagnostics, will surface.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
c99be232ad |
test(ouroboros): dump tenant-side state on bats wait failure
The smoke test at hack/e2e-apps/ouroboros.bats waits for the kubernetes-${cluster}-ouroboros HelmRelease to reach Ready and exits with `wait: timed out` on failure. cozyreport collects host-cluster state only — a tenant-side proxy or controller failure surfaces as opaque "0 of 2 pods ready" with no actionable signal, so every retry is fully blind.
Extract the tenant admin-kubeconfig before the wait (gated on the parent kubernetes HR being Ready, which is the precondition for the secret existing) and, on wait failure, dump the host HelmRelease describe, cozy-ouroboros pods, pod descriptions, recent events, controller and proxy logs (200 lines each), the upstream cozy-ingress-nginx Service and endpoints used as the proxy backend, and the tenant kube-system Corefile plus the coredns-custom ConfigMap data.
Drops the ouroboros-HR wait timeout from 25m to 20m to leave room for the diagnostic block within the bats test budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Andrei Kvapil
|
45e21a330f |
feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561)
## What this PR does Adds [`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0) — a Go reimplementation of [`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy) — and wires it into cozystack as a *conditional default*: a single `publishing.proxyProtocol: true` switch turns PROXY-protocol on at the host ingress-nginx **and** auto-deploys ouroboros to fix the resulting hairpin-NAT problem (intra-cluster traffic to the cluster's own public hostnames arriving at ingress-nginx without the PROXY header it now requires). Default stays `false` — clusters that do not run PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour change. The chart is platform-agnostic at v0.7.0: the controller and proxy binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf` inside the pod and compose the rewrite target FQDN and the backend host at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on `cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`, federations on `k8s.example.com`, custom-domain RKE2/k3s — all work without operator override. Verified end-to-end on a `cozy.local` cluster: controller logs `cluster-domain=cozy.local` at startup, chart emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked `--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy ClusterIP. Operators that prefer a chart-baked FQDN can still set `controller.clusterDomain` explicitly — the explicit-override path is preserved for non-Service backends and for split-horizon DNS topologies. The chart runs in `coredns` mode for the host (the host CoreDNS is Talos-managed and does not carry an `import` directive, so the alternative would silently no-op there) and in `coredns-import` mode for tenant deployments (tenants run cozystack-coredns, which carries the import directive and ships an empty `kube-system/coredns-custom` ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the `ouroboros.override` data key without ever touching the chart-rendered Corefile, so it does not race the chart that re-templates `data.Corefile` on every Flux reconcile). ### Layout * `packages/system/ouroboros/` — vendored chart from `oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults pinned in `values.yaml`: `controller.mode=coredns` (host scenario), `proxy.target.namespace=cozy-ingress-nginx` (override the chart-default `ingress-nginx` namespace; chart-default `serviceName=ingress-nginx-controller` matches as-is), Gateway-API watching left at upstream's default of `false`. `proxy.target.host` is left empty so the proxy composes the backend host at runtime. Image is digest-pinned at `0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c` and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is intentionally not mirrored under `ghcr.io/cozystack/*`. The chart manifest itself is also digest-pinned via `OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e` in the package Makefile, with a Chart.yaml-version vs. Makefile-version assert that aborts the update target on lockstep drift. Helm-unittest in `tests/` covers controller args, RBAC narrowing per mode, Gateway-API gating, runtime-composition flag emission in both `coredns` and `coredns-import` modes, image pin drift. * `packages/system/coredns/templates/coredns-custom.yaml` + `values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed `import` prerequisite, plus the `import /etc/coredns/custom/*.override` plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts` mount at `/etc/coredns/custom/`. Same package serves host and tenant CoreDNS, so this covers both layers without duplication. Helm-unittest pins the rendered Corefile via a snapshot so future `make update` does not silently drop the import directive. The empty rendered ConfigMap deliberately omits `data:` — Helm three-way merge has no template-side keys to compare against, so apiserver-side runtime writes (the `ouroboros.override` data key) survive every chart upgrade. The `notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the invariant. * `packages/core/platform/sources/ouroboros.yaml` — new `cozystack.ouroboros` PackageSource for the host install (`dependsOn: cozystack.networking, cozystack.ingress-application`). * `packages/core/platform/sources/kubernetes-application.yaml` — `kubernetes-ouroboros` component registered against `system/ouroboros` so the tenant HelmRelease can reference it. * `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` — tenant-side HelmRelease deployed via the tenant admin-kubeconfig, mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues` helper flips `mode` back to `coredns-import` for tenants and mirrors per-tenant `addons.gatewayAPI.enabled` into the controller (forcing `--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the controller on `WaitForCacheSync`). `controller.clusterDomain` is intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from `/etc/resolv.conf` and composes the rewrite target at runtime, so the same wrapper renders correctly on `cluster.local`, `cozy.local`, or any custom tenant cluster-domain. Operators on a non-default cluster-domain can still override via `valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a conditional `gateway-api-crds` entry on the same gate. If `addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled` is false, the HelmRelease fails the render with a clear error rather than silently skipping — the hairpin fix has nothing to fix against without ingress-nginx, and a no-op addon that quietly does nothing is harder to debug than an explicit render-time error. The valuesOverride gateway-api guard accepts every truthy spelling Helm/YAML emit (bool `true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML `yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via `toString`-then-`lower`-then-string-compare; direct bool `eq` would error on type mismatch when an operator passes the value as a string from `--set`, and case-insensitive folding catches `--set ...=On` / `--set ...=True` that pure-lowercase comparison would miss. Tenant disable is single-step: flipping `addons.ouroboros.enabled: false` deletes the rendered HelmRelease, helm-controller runs `helm uninstall` on the tenant, and the chart's pre-delete cleanup hook (`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the `ouroboros.override` key on the tenant's `kube-system/coredns-custom` ConfigMap automatically. No render-time guard is wired on the tenant side — gating on a `lookup` for the leftover HelmRelease would deadlock the routine disable (the parent render runs before helm-controller applies the missing-child diff, so the lookup would always find the leftover at the moment of the flag flip). If the chart's hook fails to land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job timeout), operators recover manually via the documented tenant cleanup recipe. * `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef with `enabled` + `valuesOverride`. `make generate` re-emitted README, JSON schema, the Go struct + deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema in lockstep. * `packages/core/platform/values.yaml` — new `publishing.proxyProtocol` flag with extensive comment covering the upstream-LB precondition (the L4 LB in front of ingress-nginx must already be injecting PROXY-v1 headers before the flag flips on, otherwise external traffic breaks), the host-vs-tenant fan-out (the flag does not cascade into tenants), the Talos re-render flap window, and the asymmetric disable path. The host disable path: flipping back to `false` reverts ingress-nginx PROXY-protocol config (flip the LB off first), but the host ouroboros itself stays installed because its Package CR carries `helm.sh/resource-policy: keep` — an explicit `kubectl delete package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages` entry) actually triggers helm uninstall, at which point the chart's pre-delete cleanup hook fires and patches the host Corefile automatically. The values comment ships a manual `sed`-based cleanup recipe as a fallback for the rare case where the chart's hook fails to land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off a `lookup` against the leftover Package CR to catch operators who flip the flag back without confirming the cleanup landed. * `packages/core/platform/templates/bundles/system.yaml` — flips between the default-form and explicit-form `cozystack.ingress-application` Package emission so the host ingress-nginx gets `use-proxy-protocol` + `real-ip-header=proxy_protocol` injected only when the flag is true (`use-forwarded-headers` and `compute-full-forwarded-for` are deliberately *not* on the list — they would let any upstream proxy spoof `X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`. Host Gateway-API watching is left at the upstream chart's default of `false` because cozystack does not install Gateway-API CRDs cluster-wide on the host. * `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in tenant, controller pod running, and after creating a TLS Ingress the rewrite snippet shows up in the import ConfigMap; an in-tenant `dig` from a netshoot pod also confirms CoreDNS actually serves the rewrite — pod-readiness alone passes vacuously since the upstream chart ships no readiness probes). ### Why not KEP-1860 `ipMode=Proxy` Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB frontend entirely when `ipMode != VIP`, breaking Service ingress for L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters fronted by an external CCM-managed proxy LB. ouroboros is the correct fix for L2/BGP-announce topologies that cozystack ships by default. ### Coverage Covered by helm-unittest (host chart, coredns wrapper, tenant addon — including the `kubernetesProvider` mock that exercises the lookup-driven `proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil CI/dry-run path, the gateway-api truthy-spelling acceptance across `true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that `--proxy-service-name` + `--proxy-service-namespace` are emitted instead of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import` modes, the assertion that `--target-service-name` + `--target-service-namespace` are emitted for the proxy backend, and the `_namespace.etcd`-empty render gate) and a tenant e2e bats test that creates a real Ingress, asserts the rewrite snippet lands in the import ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry loop tolerating CoreDNS reload latency). The host-level `publishing.proxyProtocol: true` path is not e2e-tested in this PR — it requires a PROXY-protocol-aware upstream LB, which the cozystack e2e harness does not stand up. That path is exercised manually and by docs review. End-to-end runtime auto-detect was verified manually on a `cozy.local` cluster (dev17): chart deployed at v0.7.0, controller logged `cluster-domain=cozy.local`, the rewrite block landed in `kube-system/coredns` Corefile pointing at `test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig <ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first attempt, and the chart's pre-delete hook restored the Corefile cleanly on `helm uninstall`. ### Supply-chain note Image and chart are pulled from a contributor's personal namespace (`ghcr.io/lexfrei/...`). Existing precedent (`packages/system/etcd-operator/Makefile` pulls from `ghcr.io/aenix-io/...`) is similar but `aenix-io` is the cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer call. Pairs with cozystack/website#527 (operator docs). ### Release note ```release-note feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with configurable enablement and tenant overrides. * Added a packaged Ouroboros chart providing proxy, controller, CoreDNS integration, external‑DNS and etc‑hosts modes. * Introduced PROXY‑protocol gating with an explicit acknowledgement flag. * **Documentation** * Expanded addon reference and platform docs with detailed Ouroboros/PROXY‑protocol configuration and defaults. * **Tests** * Added extensive unit and e2e tests covering rendering, RBAC, uninstall cleanup, and DNS behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
32e75ab183 |
Merge branch 'main' into daniil/split-vminstance
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
Myasnikov Daniil
|
209fa08737 |
test(e2e): decouple vminstance test from vmdisk test
"Create a VM Instance" referenced the VMDisk created by "Create a VM Disk" via the shared name "test". This implicit ordering meant any defensive pre-clean of vmdisks in test 2 would destroy the disk test 1 just provisioned, and the vm-instance chart's lookup-or-fail on the DataVolume would then surface as "Specified disk not exists in cluster: test". Make each test self-contained: - "Create a VM Disk" cleans up its own VMDisk at the end. - "Create a VM Instance" creates its own VMDisk inline (renamed to "vmi-test" to avoid colliding with any in-flight finalizer drain on test 1's resources) and waits HR -> DV -> PVC before applying the VMInstance. Replaces the pre-delete approach from the prior followup commit, which broke the data dependency. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
585ea7e089 |
test(e2e): smoke-test the ouroboros tenant addon
Auto-discovered hack/e2e-apps/ouroboros.bats spins up a tenant
Kubernetes CR with addons.ingressNginx + addons.ouroboros enabled
and asserts the cozystack-side wiring delivers a working hairpin
fix end-to-end:
* the cozy-ouroboros HelmRelease in the tenant's host namespace
reaches Ready (chartRef + admin-kubeconfig wiring is correct);
* the cozystack-coredns wrapper rendered the kube-system/coredns-custom
ConfigMap inside the tenant (the import-glob prerequisite is in
place);
* the controller pod is running in cozy-ouroboros inside the tenant;
* after creating an Ingress with a TLS hostname, the tenant's
coredns-custom ConfigMap eventually carries a `rewrite name <host>`
line under the ouroboros.override key — proves the controller is
reconciling, not merely Pod-Ready (the upstream chart ships no
readiness probes, so a controller stuck on WaitForCacheSync would
pass a pod-readiness check vacuously; the rewrite assertion catches
every cozystack-specific failure mode the unit tests cannot — wrong
namespace, missing CRDs, narrowed RBAC, gateway-api gate misfires).
End-to-end PROXY-protocol header roundtrip (the actual hairpin fix
in action) is covered by the upstream chart's own e2e on every push;
this file is the cozystack-side integration test that the cozystack
glue (chart vendoring, kubernetes-application registration, tenant
HelmRelease wiring, gateway-api gating, coredns-custom prerequisite)
holds together.
cozystack hack/cozytest.sh is plain shell, not real bats, so kubectl
output is captured directly without the `run` helper.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Myasnikov Daniil
|
cc7551cf1b |
test(e2e): use $name in kafka.bats kubectl wait (review feedback PR #2554)
Address lexfrei's nit on PR #2554: the pre-existing
`kubectl wait kafkas -n tenant-test test ...` on line 47 hardcoded the
resource name, while the existence backstop one line above was already
parametrized with `$name` in
|
||
|
Myasnikov Daniil
|
fe138afab7 |
test(e2e): add --ignore-not-found to vminstance.bats end-of-test deletes
Symmetrize the post-success cleanup on lines 83-84 with the pre-clean shape on lines 12-13/41. Without --ignore-not-found, a race that already removed the resource (e.g., a parallel teardown or finalizer-driven GC) turns a passing test into a failure after assertions have already run. Review feedback PR #2555 (lexfrei follow-up #1). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
25180c58f7 |
fix(virtual-machine): bump HR readiness wait from 5s to 60s
5s was too tight under runner load — both vm-disk-$name and vm-instance-$name HR waits could miss the operator's reconcile window. Aligns with the other timeout bumps in this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
12ccc53d9d |
test(e2e): tighten event-driven backstops
- kafka.bats: use $name in the kafkas existence backstop, matching the variable used in the kubectl wait below it (consistency with the rest of this PR's backstops). - e2e-install-cozystack.bats: the Tenant→HelmRelease materialisation is asynchronous (cozystack-controller observes the Tenant CR and creates the HR), so guard hr/tenant-test with the same existence backstop pattern used elsewhere. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
