<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
* Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The backup round-trip test could wedge for 20 min and then report a bare
EtcdSnapshot phase=Started with no root cause: dump_diagnostics dumped the
(healthy) EtcdCluster and the operator log, but the snapshot agent runs in
its own Job Pod, so the etcd-read / S3-upload error lives only in that Pod's
log and in the BackupJob/EtcdSnapshot status.
- dump_diagnostics now dumps the whole backup chain: BackupJob/Backup and
EtcdSnapshot describe, each snapshot Job's describe + Pod logs (selected by
the operator's etcd-operator.cozystack.io/cluster label), and recent
namespace events (the primary signal when a hung Pod logs nothing).
- wait_for_field gains an optional terminal-failure value; steps 04/05 pass
Failed so the BackupJob/RestoreJob waits return immediately on a terminal
phase instead of polling to the timeout — which also keeps the snapshot
Job Pod within its 600s TTL so its log is still readable at dump time.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The v1alpha2 operator sets Available=True early (reason ClusterDiscovered)
as soon as the seed member forms the cluster, then scales the remaining
members up one at a time. Gating the runtime-contracts test on Available
raced the member-Pod count assertion and saw only 1/3 Pods, failing the
run. Gate on status.readyMembers=3 instead — the count of members whose
EtcdMember Ready condition is True (and the /scale status path), which
only reaches 3 once every member Pod is Running and serving. Bump the
wait timeout to 300s to cover sequential one-at-a-time rollout.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats: read the container name, metrics port, and Pod IP with the strict member selector (name+instance+managed-by) plus a Running-phase filter instead of the broad app.kubernetes.io/name=etcd, so .items[0] cannot resolve to a non-member or not-yet-running Pod. Also assert a non-empty Pod IP before the metrics probe.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
kubernetes-previous.bats intermittently fails on
`kubectl wait deploy --for=condition=available` with
`Error from server: etcdserver: leader changed` even when the target
Deployment is milliseconds away from becoming Ready. Observed in run
28772259934 on head `5f0bec622`: three of the four deployments
already reported `condition met`, the fourth (the Kamaji tenant CP
Deployment) was ~200ms from Ready per the crust-gather snapshot
captured 20s later (`readyReplicas: 2`), and the wait exited
non-zero on the FIRST etcd-side error it saw on the watch channel.
Cause: the e2e sandbox management cluster is a 3-node kind on Talos
VMs with a 3-instance etcd HA cluster. By the time
kubernetes-previous starts (after kubernetes-latest,
kubernetes-oidc-customconfig, kubernetes-oidc-system, plus the
storageclass-fallback probe), accumulated CDI+DRBD IO pressure and
lingering "still attached to node" PVCs from prior tests push etcd
fsync latency high enough to trip heartbeat timeouts on the leader.
The election that follows resolves in seconds, but `kubectl wait`
sees the server-side error mid-watch and exits — clients would
normally reconnect, but `wait` treats a watch-channel error as
terminal.
Wrap the four management-cluster `kubectl wait` calls in
run_kubernetes_test with a retry helper that:
- Retries only on a curated allowlist of server-side transients
(`etcdserver: leader changed`, `etcdserver: request timed out`,
`the server was unable to return a response in the time
allotted`).
- Never swallows legitimate errors (NotFound, --timeout expired,
permission denied, etc.).
- Caps at 3 attempts with a 5s backoff so a persistently-broken
etcd still surfaces loudly.
Untouched:
- `kubectl wait namespace tenant-test --timeout=20s` at line 308 —
short-lived, hits mgmt but the object always exists ~instantly.
- `kubectl wait deployment ... --kubeconfig tenantkubeconfig-...` at
line 509 — waits on the TENANT apiserver's own etcd, a different
failure domain (single-instance kine or CNPG-backed via Kamaji,
not the 3-instance mgmt etcd that flapped).
- Other bats files with mgmt-cluster `kubectl wait` — out of scope
for this PR; if the same flake reappears there a follow-up can
lift the helper into a shared library.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The audience-list assertion used \`grep -qF "- tenant-test-..."\`.
Because the pattern begins with a dash, grep treats the leading \`-\`
as an option flag, aborting the test with \`grep: invalid option --
' '\` before the actual match ever runs. Insert an explicit \`--\`
end-of-options separator so the YAML list-item pattern is passed as a
pattern rather than a flag.
Custom-config counterpart is unaffected — its grep patterns do not
start with a dash.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
cozytest.sh (the e2e runner) is not real bats — it never invokes
setup()/teardown(). The OIDC files were structured around those hooks
to seed TEST_NAME and cleanup the tenant CR between tests, so with
cozytest strict-mode wrapping (\`set -eu\`), the first \${TEST_NAME}
expansion inside an @test aborted with 'parameter not set' before the
apply could even run.
Rework both files to the convention documented in external-dns.bats:
seed TEST_NAME at file scope (no per-test uniqueness needed — the
tenant-test namespace is single-suite), keep cleanup_kr() as a shared
helper, add a file-level cozy_cleanup() wrapper (the hook cozytest
actually calls on suite exit / first-failure), and move the pre-test
CR delete inline at the top of the first @test that creates the CR.
Subsequent @tests only read state and rely on the CR persisting until
cozy_cleanup.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Three items from the gemini-code-assist and coderabbitai passes that
survived the Phase-1 redesign:
- `oidc-rbac-job.yaml` (gemini HIGH): pre-delete cleanup uses
`kubectl --request-timeout=30s` in addition to `--timeout=30s`
on the delete verb, so an unreachable tenant apiserver cannot
stall the pre-delete hook on the HTTP round-trip itself. The
`|| echo WARNING` fallback still keeps a truly unreachable
cluster from blocking `helm uninstall`.
- `oidc-keycloak.yaml` + `_helpers.tpl` (gemini LOW): switch
`_cluster.{oidc-enabled,root-host}` reads to `dig` with an empty
map default so a missing `_cluster` key nil-safely emits an
empty string and hits the platform-OIDC guard instead of a
template panic on `.Values._cluster`.
- `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
(coderabbit MAJOR): add an existence-poll `timeout ... until
kubectl get secret ... ; do sleep ...; done` before the
`kubectl wait secret --for=jsonpath` line. Without it a runner
that reaches the wait before the HR has rendered the Secret
gets an immediate `NotFound` and the test flakes.
Adds one helm-unittest case covering the nil-safe `_cluster` path.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the design proposal explicitly lists TWO bats files:
`kubernetes-oidc-system.bats` and `kubernetes-oidc-customconfig.bats`.
The first revision shipped only one combined file; this commit splits
them.
The CustomConfig bats covers what System cannot: the
operator-supplied AuthenticationConfiguration is plumbed onto the
KamajiControlPlane, the resulting Secret carries the BYO issuer (and
emphatically NOT a `cozy` issuer), nothing lands in the `cozy` realm,
and no OIDC kubeconfig Secret is pre-rendered (the operator
distributes their own).
The System bats header is updated to call out the deferred parts
(in-tenant CRBs / view-vs-admin RBAC behavior / browser-flow login)
that need a full Kamaji bringup, which is already covered for the
cluster path by `kubernetes-latest.bats`.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the design proposal specifies
`name: authentication-config` and
`mountPath: /etc/kubernetes/authentication-config`, but the previous
revision had renamed both to `cozy-oidc-authn-config` and
`/etc/kubernetes/cozy-oidc` to dodge a collision with the volume
name used in `apiserver_authentication_test.yaml`. That test was
demonstrating the GENERIC user-supplied `extra*` passthrough — not
the OIDC use case — so rename the test fixture instead and restore
the design-spec names on the chart side.
The reservation guard against user-supplied volumes reusing
`authentication-config` only fires when `spec.oidc.mode` is not
None, so clusters with OIDC off retain the previous freedom to name
their own volumes anything.
Renames in `apiserver_authentication_test.yaml`: the user-supplied
volume name `authentication-config` becomes `tenant-byo-mount`; the
Secret name `tenant-authentication-config` and the apiserver flag
`--authentication-config=` are untouched.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fixes for four linked lifecycle issues:
- The post-install Job created `<release>-oidc-kubeconfig` via
`kubectl apply` from inside the script; the resulting Secret had
no Helm release labels and was orphaned on `helm uninstall`. Now
a release-owned placeholder Secret is rendered up front in
`oidc-kubeconfig-secret.yaml` and the Job patches its `data`
field at runtime (the tenant CA still has to come from the
Kamaji-issued admin kubeconfig, which appears asynchronously).
- The ServiceAccount/Role/RoleBinding for the bootstrap Job were
hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
so they leaked on `helm uninstall`. They are now regular release
resources (no hook annotations) and are reaped by `helm uninstall`
alongside everything else. Both the post-install bootstrap Job
and the pre-delete cleanup Job find a working SA.
- The previous design tried to add `hook-succeeded` to the SA, but
that would delete the SA right after the bootstrap hook
succeeded — leaving the pre-delete cleanup Job with no SA to
schedule. Release-owned resources sidestep this race entirely.
- The Role had `verbs: ["create"]` on `secrets` without
`resourceNames` (Kubernetes RBAC cannot scope `create` by name).
With the placeholder Secret pre-rendered the Job only needs
`get`/`patch` on the specific name — the broad `create` rule is
removed.
Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.
Adds three helm-unittest cases for the placeholder Secret across all
three modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Validate on a live cluster that:
- the new `spec.oidc.*` fields are admitted by the Kubernetes CRD;
- cozystack-api accepts the updated schema and renders a HelmRelease;
- the KamajiControlPlane gains `--authentication-config` and the
chart-owned `cozy-oidc-authn-config` mount and volume;
- the AuthenticationConfiguration Secret carries the cozy realm
issuer URL and the per-cluster audience;
- per-cluster `KeycloakClient` + audience `KeycloakClientScope`
land in the cozy realm (with the EDP operator present).
The full end-to-end RBAC flow (Job → CRBs inside the tenant cluster)
needs a Kamaji bringup, which kubernetes-{latest,previous}.bats
already cover for the cluster path; the OIDC browser flow is
explicitly out of scope per the design proposal. This file covers
only what the render-side regression budget can pay for.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
kubernetes-previous.bats intermittently fails with `zfs create -V ...
returned with exitcode 1: cannot create '...': out of space` on a
satellite even though the LINSTOR pool measured 30+ minutes later shows
100+ GiB free per satellite. Root cause proven from a satellite
ErrorReport captured in the cozyreport artefact (PR #3044 run
28751310913, report id 6A4AADFD-349B2-000000): the previous tenant
Kubernetes test's cozy_wait_tenant_drained gate returned as soon as
its PVCs disappeared from the API, but the satellite-side ZFS `zvol
destroy` for the just-deleted replicated (autoPlace=3, DRBD) 20 GiB
worker disks lagged tens of seconds behind that signal as DRBD
adjusted, unref counts drained, and ZFS batch-destroyed the datasets.
The next tenant test's CreateVolume then landed inside that window
against a pool that had not yet reclaimed the space, LINSTOR-CSI
retried autoplace against its own still-being-torn-down previous
placement, and worker Machine bringup stretched past the MHC
nodeStartupTimeout budget with no cluster-side fault.
Add a companion `cozy_wait_linstor_pool_free` helper that polls the
LINSTOR controller for the smallest FreeCapacity across every ZFS
satellite pool and blocks until it reaches a per-satellite threshold
(default 90 GiB — covers the ~82 GiB worst-case footprint of two
20 GiB replicated worker targets across all three satellites plus two
21 GiB CDI scratch PVCs landing on the same node via the local
storageClass). Called from cozy_cleanup immediately after
cozy_wait_tenant_drained. Bounded and best-effort: 300 s timeout,
`|| true` at the call site, warning printed with the observed
pool state if the threshold is not reached in budget so the next test
does not misdiagnose the pressure.
The threshold is a floor for the concrete failure mode observed in the
E2E suite; the surrounding kubernetes-latest -> kubernetes-previous
handoff is the only run in the current suite that immediately follows
a heavy tenant teardown with another tenant provisioning, so this
gate is the natural place for the check. E2E-scope only, no product
or chart change.
The proximate observation (`zfs create out of space`) surfaced because
of PR #3209 which taught hack/cozyreport.sh to bundle satellite
ErrorReports; without that artefact the root cause was ambiguous
between multiple plausible LINSTOR internal races. Ships alongside
that PR.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The Gateway (cozystack-gateway-hostname-policy) and Route
(cozystack-route-hostname-policy) admission policies constrain tenant
hostnames on the Gateway API path. Gateway API is opt-in and off by
default, so in the default configuration tenant applications publish
through a legacy networking.k8s.io/Ingress on the shared ingress-nginx,
which had no equivalent policy.
Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in
tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[]
is allowed when it is within the namespace's own apex
(namespace.cozystack.io/host), or when it is a concrete (non-wildcard)
name entirely outside the platform root apex (_cluster.root-host); a
hostname under the platform root apex but outside the namespace's own
apex, and a wildcard taken through the outside-root path (which could
match the platform apex itself), are rejected. The outside-root branch
keeps the kubernetes app's Proxied exposeMethod working, since it routes
a user-supplied external domain to a nested cluster. A rule with no host
(absent or empty, an unbounded catch-all) and spec.defaultBackend are
also rejected.
The policy templates the platform root apex into its CEL and renders
only when _cluster.root-host is set, so it is never shipped in a state
that would admit every hostname. A tenant-* namespace missing its own
apex label is denied.
Cover the policy with helm-unittest (render, including the root-host,
empty-host, and wildcard guards) and behavioral admission tests against
the live apiserver (in-apex allow, external-domain allow,
own-apex-wildcard allow, external-tls allow, under-root-outside-apex
deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny,
hostless deny, empty-string-host deny, defaultBackend deny), and document
it as an additional layer in the gateway security model.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.
Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:
- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
external access via `service.type: LoadBalancer` and keep `primaryService`
LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
(no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
-primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
installs are visible in the dashboard.
Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
When a tenant Kubernetes cluster hits the 12m node-Ready deadline with
fewer than two nodes joined, the failure path dumped only `describe
nodes` and `get hr`. That cannot tell apart the two node-join failure
sub-modes: a worker VM that never booted (virt-launcher Pending or
OOMKilled) versus a VM that booted fine but whose kubelet never
registered a Node. The tenant cilium-operator HelmRelease reports
"InProgress" identically in both, so the current dump is not enough to
diagnose the failure or design a fix.
Dump scoped, guarded diagnostics before failing fast:
- worker VM/VMI/virt-launcher state on the management cluster, which is
what actually splits the two sub-modes;
- tenant kubelet CSRs plus the talos-csr-signer sidecar log, which reveal
a boot-but-no-Node stall.
In-guest Talos/kubelet capture is deliberately left out: talosctl needs a
tenant-cluster talosconfig the runner does not have (the tenant workers
use a separate Talos CA from the management sandbox), so the gap is
documented in the failure path rather than shipped as a call that would
fail mTLS and capture nothing.
Every capture is guarded with `|| true` so a capture failure never masks
the real exit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The inter-test cleanup hook deletes the tenant Kubernetes CR and waits
only for the CR to disappear. That wait finalizes as soon as the CR's
finalizers clear, which merely TRIGGERS teardown of the CAPK worker VMs
and their DataVolume-backed disk PVCs -- it does not await it. The
virt-launcher pods keep their guest RAM reserved until the VMIs are
gone, so the next back-to-back tenant test's worker VMs begin scheduling
against a sandbox the previous tenant has not yet vacated. Under the
constrained CI sandbox this starves memory, a worker VM misses the
node-join budget, and the run flakes on worker-node-join (the
MachineHealthCheck cannot remediate when both workers are unhealthy).
Block cozy_cleanup until the tenant VMs, VMIs and disk PVCs have
actually drained from tenant-test before it returns, so the next test
starts on a freed sandbox. The wait is bounded (5m) and best-effort:
cozytest runs cozy_cleanup wrapped in `|| true` and the waiter returns
loudly on timeout, so a stuck teardown can never hang the job longer
than the deadline. The probe is namespace-scoped -- tenant-test is
provisioned with etcd/monitoring/seaweedfs disabled so it carries no
baseline PVCs, and e2e apps run sequentially each cleaning up after
themselves, so the only VMs/VMIs/PVCs present belong to the tenant being
torn down.
The pure drain exit-condition (cozy_tenant_drained) is unit-tested in
hack/run-kubernetes-drain_test.bats, including the invariant that a
failed probe's "err" sentinel is never misread as a completed drain.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Every `*.bats` file in the repo runs through `hack/cozytest.sh`, not
real bats — the e2e files via `packages/core/testing/Makefile` and the
`hack/*.bats` unit tests via `make bats-unit-tests`. cozytest runs each
`@test` under `set -e`, and POSIX errexit is suppressed for a pipeline
beginning with `!`. So a bare `! cmd` must-not-happen assertion never
fails the test: when the forbidden condition actually occurs (the
command succeeds), the negated pipeline returns non-zero but errexit
ignores it and the test passes anyway. Ten such assertions across six
files were silently vacuous.
This converts each to `if cmd; then echo "<reason>"; false; fi`. The
`if` condition is exempt from errexit (so the probe itself never aborts
the test), and the `false` in the taken branch is a plain command under
`set -e`, so it aborts and fails the test with a legible message when
the forbidden condition holds. This is the must-not-happen inverse of
the `cmd || { echo …; false; }` pattern already used in the suite (e.g.
harbor.bats, per docs/agents/e2e-testing.md). Original redirections and
quoting are preserved, so happy-path behavior is unchanged — only the
failure path now actually fails.
Converted assertions:
- `bucket`: a read-only user must not be able to upload (access-control
regression guard)
- `gateway`: a child tenant must not have its own gateway HelmRelease
(inheritance)
- `kuberture`: external-dns public/internal hostname isolation
- `e2e-install-cozystack`: tenant-name validation must reject rather
than warn; the version ConfigMap must be gone after delete
- `build-matrix` (unit test): talos/installer must be excluded from the
parallel matrix
- `capture-dataplane` (unit test): kube-dns and pending rows must be
filtered out
A separate in-flight etcd test change owns two more such assertions in
`etcd.bats`; they are left out of this PR to avoid a conflicting edit.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Strengthened several end-to-end/Bats assertions to fail more
explicitly when expected cluster conditions aren’t met.
* Improved matrix validation to reliably confirm excluded components are
absent.
* Updated dataplane load balancer filtering, bucket readonly upload, and
gateway inheritance checks to avoid false/ambiguous pass cases.
* Tightened DNS hostname isolation probes and cozystack installer
validations (tenant name dashes, deletion protection) with clearer
“expected vs found” failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
These bats files are all run by hack/cozytest.sh, which executes each
@test under `set -e`. POSIX errexit is suppressed for a pipeline that
begins with `!`, so a bare `! cmd` must-not-happen assertion never fails
the test: when the forbidden condition actually occurs (cmd succeeds) the
negated pipeline returns non-zero but errexit ignores it and the test
passes anyway. These assertions were silently vacuous.
Replace each with `if cmd; then echo "<reason>"; false; fi`, the idiom
already used and documented elsewhere in these files — the if-condition
is exempt from errexit and the `false` in the taken branch fails the test
loudly. The forbidden condition now fails the test as intended:
- bucket: read-only user must not be able to upload (access control)
- gateway: child tenant must not have its own gateway HelmRelease
- kuberture: external-dns public/internal hostname isolation
- e2e-install: tenant-name validation must reject (not warn); the version
ConfigMap must be gone after delete
- build-matrix: talos/installer excluded from the parallel matrix
- capture-dataplane: kube-dns and pending rows are filtered out
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozytest.sh — the runner for every bats file here — is not real bats: it
never invokes setup()/teardown(). So these hooks were dead. Cleanup ran
only via each test's success-path inline delete, and a failing test
leaked its resources into the shared tenant-test namespace (and, for
kuberture, cluster-scoped RBAC that persists across sandbox re-runs).
Wire the cleanup to cozy_cleanup(), the hook cozytest does run at suite
exit and on the first failing test:
- external-dns: dead teardown() -> cozy_cleanup() deleting all ExternalDNS
CRs. ExternalDNS is a stateless Deployment with the inmemory provider,
so a plain delete fully reclaims it — no drain needed.
- kuberture: dead teardown() -> cozy_cleanup() routed through the existing
_cleanup(), so the probe Pods, SA, and cluster-scoped ClusterRole and
ClusterRoleBinding are reclaimed on the failure path too.
build-matrix_test: drop the dead setup() entirely — it only cd'd to the
repo root, which cozytest already runs from, so the relative
hack/build-matrix.sh calls resolve without it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
After deleting a `Qdrant` app CR, the StatefulSet data PVC
(`qdrant-storage-<release>-0`) was left orphaned in the tenant
namespace,
consuming storage permanently. StatefulSet-templated PVCs are never
garbage-collected on StatefulSet deletion by default.
The vendored upstream Qdrant chart does **not** support
`persistentVolumeClaimRetentionPolicy` (verified: its `statefulset.yaml`
never
renders that field), so the OpenBAO-style value passthrough is not
possible
here. Instead this adds a `post-delete` PVC cleanup hook (modeled on the
MariaDB
hook) that deletes the release's PVCs by label.
The chart's volumeClaimTemplate only stamps `app: qdrant` (shared by
every
Qdrant in a namespace), so to select PVCs safely per release this also
passes
`persistence.additionalLabels.app.kubernetes.io/instance: <release>`
through the
HelmRelease — giving the data PVC a release-unique label the hook
selects on.
The cleanup Job is hardened: runs as non-root (uid 65534), read-only
root fs,
all capabilities dropped, `seccompProfile: RuntimeDefault`; RBAC scoped
to
`persistentvolumeclaims`. A real cleanup failure is logged (not silently
swallowed) but never blocks namespace teardown.
> [!WARNING]
> **Deleting a Qdrant app now permanently removes its data PVC and the
vector data it holds.**
> This is intentional and fixes the storage leak; the hook only fires on
full
> release deletion (not on scale-down). Back up before deleting.
Fixes#3059.
### Release note
```release-note
fix(qdrant): garbage-collect the StatefulSet data PVC when a Qdrant app is deleted, instead of leaving it orphaned. WARNING: deleting a Qdrant app now permanently removes its data PVC and the data it holds.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a post-delete cleanup hook for Qdrant to remove leftover
StatefulSet data PVCs after deletion.
* Included hardened Kubernetes settings for the cleanup job to improve
runtime safety.
* **Tests**
* Added Helm chart tests that validate the post-delete cleanup job and
its RBAC permissions.
* Enhanced end-to-end verification to confirm the data PVC is fully
removed after deleting a Qdrant instance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.
The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.
The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.
Implements the attachment model from #1614.
### Release note
```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
* Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The post-delete hook selected PVCs with an unanchored name prefix
(qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting
release "qdrant-test" matched not only its own PVC
(qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC
(qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting
the sibling's data (or wedging it in Terminating via pvc-protection).
Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own
data PVCs match. A release name that is a prefix of a sibling no longer
collides.
- cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$".
- tests: assert the anchored selector and add a notMatchRegex guard against
regressing to the unanchored prefix.
- e2e: gate the post-delete wait on a NotFound result specifically, so a
transient API/auth error is not misread as "PVC reclaimed".
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
## What this PR does
The e2e runner `hack/cozytest.sh` is not real bats — it converts `@test`
blocks to shell functions and recognises a `cozy_cleanup()` hook, but
never calls `setup()`/`teardown()` and never opens fd 3. So
`etcd.bats`'s `setup()` never ran (etcd leaked its HelmRelease and data
PVCs into the shared `tenant-test` namespace) and `dump_diagnostics`
wrote to an unopened fd 3, silently dropping failure diagnostics. The
teardown's `kubectl wait --for=delete` lines also passed
`--ignore-not-found`, which `wait` does not accept.
This moves teardown into an `etcd_drain()` helper, run three ways: a
pre-clean at the start of the first scenario, inline at the end of the
last scenario (under `set -e`, so a teardown that does not settle fails
the suite), and as `cozy_cleanup` (the best-effort safety net cozytest
runs at suite exit and on the first failing test). The drain reclaims
what a plain CR/HR delete leaves behind:
- the StatefulSet's retained `data-etcd-*` PVCs (not owned by the Etcd
CR) — a later install otherwise reuses the same-named volumes, boots the
previous member identity, never forms a fresh quorum, and the chart's
`helm --wait` fails; and
- the cluster-scoped Kamaji `DataStore`, whose leaked finalizer wedges
the HR uninstall — cleared each pass until the Etcd CR, HR, DataStore,
and PVCs are all gone, bounded by a 4-minute deadline that fails the
drain on timeout.
Remnants are probed by name so a transient API error is never mistaken
for a successful delete. Diagnostics now go to stdout, which cozytest
captures and prints on failure.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved end-to-end cleanup so leftover resources are removed more
reliably between runs.
* Made failure diagnostics easier to capture when tests fail.
* Added stronger cleanup handling at the start and end of the test flow
to reduce flaky suite behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Lays the foundation for migrating tenant LoadBalancer exposure off the
deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off
MetalLB onto Cilium-native announcement — the mechanism-agnostic shape
prescribed when the earlier chart-side approach (#2468) was reverted in
#2603.
It adds a new `network.cozystack.io/v1alpha1` API group and a controller
that translates intent into whichever LoadBalancer backend the cluster
admin configured:
- `ExposureClass` (cluster-scoped, admin): binds a logical class name to
a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP
address scope — the StorageClass/IngressClass pattern.
- `ServiceExposure` (namespaced, tenant intent): "expose this Service
via class X"; status reports the resolved backend and assigned IPs.
- A `serviceexposure` controller renders the backend's pool + announcer
once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium
`CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to
the namespaces that opted in. Backends are pluggable behind a small Go
interface, so adding BGP or another cloud LB is one implementation, not
an API change. The controller never mutates the target Service — the
chart still owns its shape.
Platform wiring: `publishing.exposureClass.*` renders the admin
`ExposureClass` and the host ingress publishes through it. The default
(no class set, `externalIPs`) leaves every existing cluster's behaviour
byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes
a single `ExposureClass.spec.backend` change.
This is the foundation only — Part of #2586. Converting per-app
exposure, flipping Cilium `l2announcements` on, changing the default,
and removing MetalLB are deferred to later phases.
### Release note
```release-note
feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added new `ExposureClass` and `ServiceExposure` custom resources with
support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB),
including optional L2 and address configuration.
* Added Helm wiring and templates to enable service/ingress publishing
via `ServiceExposure`, selecting a configured (or default)
`ExposureClass`.
* **Bug Fixes**
* Improved reconciliation, finalizer-based cleanup, and status updates
for readiness and assigned IP reporting, including safer handling when
backends or managed resources change.
* **Tests**
* Added end-to-end and unit test coverage for backend rendering,
readiness, and cleanup flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The chart shipped a default nodeGroups.md0 in values.yaml. Helm deep-merges
values, so the default md0 was re-added on top of any user-supplied
nodeGroups; combined with Kubernetes stripping null values, md0 could never
be removed by configuration. Move the default into the template
("kubernetes.nodeGroups" helper) so it is applied only when nodeGroups is
empty. User-supplied node groups now fully replace the default, letting users
name and omit groups (including md0) freely.
Test fixtures that relied on the chart-default md0 to supply diskSize via
values merge now set it explicitly.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
The previous approach routed a release-unique label through
qdrant.persistence.additionalLabels, which the upstream chart renders into
the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is
immutable on an existing StatefulSet, so for any already-deployed Qdrant the
API server would reject the upgrade and the <release>-system HelmRelease
(retries: -1) would retry-fail forever — permanently wedging reconciliation.
Label changes on a volumeClaimTemplate also never apply to PVCs that already
exist, so the cleanup hook could not reclaim those instances anyway.
Switch the post-delete hook to select PVCs by the deterministic StatefulSet
naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely
from the release name, so this needs no StatefulSet change (upgrade-safe) and
also reclaims PVCs from instances created before the hook existed.
- qdrant.yaml: drop the additionalLabels passthrough entirely.
- hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep.
- tests: assert the name-prefix command instead of the label passthrough.
- e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited).
BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC
and the vector data it holds.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The e2e runner (hack/cozytest.sh) is not real bats: it never invokes
setup()/teardown() and never opens fd 3. So etcd.bats's setup() never ran
and dump_diagnostics wrote to an unopened fd 3, silently aborting on
failure — and etcd leaked into the shared tenant-test namespace.
Move teardown into etcd_drain() and run it inline at the end of the last
scenario (under set -e, so a teardown that does not settle fails the
suite) and as cozy_cleanup (the best-effort hook cozytest runs at suite
exit and on the first failing test). The first scenario also drains up
front so a dirty namespace cannot taint it.
etcd_drain reclaims what a plain CR/HR delete leaves behind:
- the StatefulSet's retained data-etcd-* PVCs (not owned by the Etcd CR);
a later install otherwise reuses them, boots the old member identity,
never forms a fresh quorum, and the chart's helm --wait fails; and
- the cluster-scoped Kamaji DataStore, whose leaked finalizer wedges the
HR uninstall — cleared each pass until the Etcd CR, HR, DataStore, and
PVCs are all gone, bounded by a deadline that fails the drain on
timeout. Remnants are probed by name so a transient API error is never
mistaken for a successful delete.
Drop --ignore-not-found from the teardown waits (not a valid wait flag);
the drain polls for existence instead. Diagnostics now go to stdout,
which cozytest captures and prints on failure.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a deterministic e2e for the externalIPs backend: it creates an
ExposureClass, a Service with pinned externalIPs, and a ServiceExposure,
then asserts the controller resolves the class, renders no MetalLB/Cilium
pool, reports the assigned IPs, and cleans up via its finalizer on delete.
The externalIPs path is chosen so the test creates no backend pool — it
cannot collide with the cluster's existing LoadBalancer address pools or
depend on real L2 reachability. The metallb and cilium rendering paths are
covered by the Go unit tests.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Pin the PR #2872 B1 fix: when the management cluster exposes only remote
LINSTOR StorageClasses whose names sort after "kubevirt" and none is named
the configured storageClass (default "replicated"), the tenant default must
be chosen among the propagated classes and never the legacy "kubevirt" alias.
helm-unittest cannot reach this branch (no live cluster -> lookup returns
empty -> the storageClasses map collapses to the "replicated" fallback), so
it is exercised in e2e via a single server-side dry-run render of the
kubernetes chart against the live management cluster: two remote LINSTOR
classes are added and "replicated" is removed for the one render, then the
rendered -csi HelmRelease storageClasses map is asserted to carry exactly one
default among the propagated classes, never the "kubevirt" alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- csi_test.yaml: assert the fallback storageClasses map yields a default
"replicated" plus the non-default "kubevirt" alias, and that the
alias tracks the storageClass selector.
- kubevirt-csi-node: add a Makefile test target (otherwise the package
is silently skipped by the unit-test runner) and storageclasses_test
covering per-entry rendering, the single-default invariant, and the
absence of any enforcement allowList in driver-config.
- run-kubernetes.sh: verify in e2e that "replicated" propagates as the
default kubevirt-CSI class, the "kubevirt" alias is retained, and the
node-local "local" infra class is filtered out.
Refs: #2094
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
> **Supersedes #2610.** Same work, rebased cleanly onto current `main`
so it now sits on top of #2867 (the `kube-controller-manager` VAP
type-checker fix). That fix is what was killing the base install in
#2610's E2E *before* the `kubernetes` app test ever ran, leaving the
Talos worker-bootstrap rewrite verified only by code reading. #2610's
67-commit branch is squashed here into 8 logical commits with original
authorship preserved (@kvaps and @IvanHunters as authors); the tree is
**byte-identical** to #2610's head — no code changes, just the rebase
onto `main` (to pick up #2867) plus history cleanup. The detailed review
history lives on #2610.
Phase 1 of the Kubernetes-app split design (see cozystack/community#8):
replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by
`cluster-api-bootstrap-provider-talos` (CABPT) and a
`clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane
pod. Existing tenants keep working — old machines roll out and get
replaced by Talos workers without manual intervention.
Highlights:
- Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the
existing kubeadm one in `capi-providers-bootstrap`.
- Bump the Kamaji control-plane provider with an upstream-bound patch
that exposes `KamajiControlPlane.spec.network.additionalServicePorts`,
used to publish trustd (50001/TCP) on the apiserver service.
- Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and
stable random Talos secrets (`token`, `clusterId`, `clusterSecret`,
`bootstrapToken`) using the helm lookup-and-reuse pattern.
- Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the
tenant `kube-system` via a Helm post-install/upgrade Job.
- Render a `TalosConfigTemplate` (worker machineconfig, `generateType:
none`) and switch the `MachineDeployment.bootstrap` reference from
`KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets
and the Kamaji apiserver service being ready.
- Boot workers from the Talos openstack image via a CDI `DataVolume`
(`source.http.url` → `factory.talos.dev`), expose the system disk as
virtio-blk with `blockSize.custom: logical=512, physical=4096` so
4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's
O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk
(Talos lays out EPHEMERAL itself).
- Pin pod/service CIDRs in the worker machineconfig to the tenant ranges
(`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap
diagnostic against the host pod CIDR.
- Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the
Kamaji apiserver Service ClusterIP (looked up at render time), so
bootstrap survives the chicken-and-egg moment where tenant DNS does not
exist yet and the host CoreDNS does not serve the tenant zone.
End-to-end on a dev cluster: a fresh tenant brings up a Talos worker,
kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving
by the talos-csr-signer sidecar), Cilium initialises with hostNetwork
against the apiserver ClusterIP, CoreDNS comes up, and the node
transitions to `Ready`.
### Release note
```release-note
feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
**New Features**
* Added Talos worker configuration for `talos.version` and
`talos.schematicID`
* Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`)
for worker remediation tuning
* Added image overrides for `kubectl` and `talosCsrSigner`
* Exposed additional tenant control-plane Service ports support
**Breaking Changes**
* Removed Kubernetes v1.30; supported versions start at v1.31
* Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with
consolidated Talos system-disk semantics
**Removals**
* Removed Ubuntu container-disk images and related containerd patch
mechanisms
* Air-gapped tenant workers temporarily unsupported during the Talos
transition
**Documentation**
* Updated chart docs, parameters, and GitOps upgrade guidance for
version bumps
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Raise the `kafka` and `zookeeper` components in the Kafka e2e test from
the `nano` preset (250m / 128Mi) to `c1.small` (1 CPU / 1Gi), the chart
default.
## Why
128Mi cannot hold the Strimzi Kafka 3.9.1 and ZooKeeper JVMs. The
brokers OOMKill on startup (exit 137) and enter CrashLoopBackOff, so the
pods never reach Ready, the `Kafka` CR and its HelmRelease never go
Ready, and the 5m readiness wait times out. It presents as a flake
rather than a hard failure because 128Mi sits right at the JVM-start
threshold — occasionally a broker becomes Ready before its RSS crosses
the cgroup limit.
`c1.small` (1Gi) comfortably fits both JVMs and matches the chart
default, so the test exercises the documented default sizing. The e2e
sandbox VMs are 8 CPU / 24Gi, so 1Gi per pod (4Gi total) across the four
pods (kafka-0/1, zookeeper-0/1) is well within budget.
## Scope
Test-only; values changed, test structure untouched. MariaDB's e2e test
also uses `nano` but was deliberately left as-is: it runs MariaDB (C) +
a Go exporter with no JVM, its chart default is already `t1.nano`
(128Mi), and it shares the `nano` preset with the passing
postgres/redis/qdrant tests — not the same OOM class.
## What this PR does
**Problem**
The `foundationdb` e2e test fails intermittently with a `.status`
jsonpath poll timeout even though the FoundationDB cluster is fully
healthy and reconciled. The `generations.reconciled`,
`hasListenIPsForAllPods`, `health.fullReplication` and `health.healthy`
gates polled their status fields with a 60s ceiling, while the
strictly-earlier `health.available` gate already allowed 300s. The
operator stamps `generations.reconciled` only after every process holds
its role for `minimumUptimeSecondsForBounce` (60s, set in
`packages/apps/foundationdb/templates/cluster.yaml`), which makes
`reconciled` — and the health fields gated on it — structurally the last
status fields to converge. The short ceiling closed the gate seconds
before the field landed; a snapshot taken a few seconds after a failure
shows the live object fully reconciled (`generations.reconciled: 1`,
`health.healthy: true`).
**Fix**
Raise those four late-converging gates to 300s, matching the existing
`health.available` budget. The waits stay event-driven (`until …; do
sleep …; done`) — only the ceilings widen, no new retries or fixed
sleeps. A comment records why `reconciled` converges last so the budget
ordering stays legible to future readers. The early-converging 60s gates
(`configured`, `connectionString`, `databaseConfiguration.*`,
`desiredProcessGroups`) and the spec-field gates are left untouched,
since they settle before `health.available` and 60s is already adequate.
**Why not just rerun**
Rerunning masks a deterministic ordering bug: the latest-converging
field carried the tightest budget. The cluster was healthy on every
"failure" — the test was racing its own success criteria, so the failure
recurs until the budget ordering is corrected. This change touches only
the test harness; the platform behaviour it asserts is unchanged.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased waiting time in one end-to-end FoundationDB setup test so
cluster status checks have more time to stabilize.
* Reduced flaky failures by allowing key health and reconciliation
fields to converge before the test continues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Rewrite the SecurityGroup e2e for the membership model: assert the backing
CiliumNetworkPolicy's endpointSelector is the group's own membership label,
that attachments ride a storage-owned annotation hidden from the view, and
that fromApp/fromSG peers project to lineage- and membership-label endpoint
selectors the real cilium.io/v2 CRD accepts. A second test creates a
managed-labeled pod and asserts the securitygroup-controller stamps the
membership label on attach and clears it on detach.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The kafka e2e test pinned both the kafka and zookeeper components to the
legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the
Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup
(exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka
CR and its HelmRelease never go Ready, and the 5m readiness wait times
out. It surfaced as a flake because 128Mi sits right at the JVM-start
threshold — occasionally a broker passes readiness before its RSS crosses
the cgroup limit.
Raise both components to "c1.small" (1 CPU / 1Gi), the chart default,
which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so
1Gi across the four pods is well within budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The reconciled, hasListenIPsForAllPods, fullReplication and healthy gates
polled status fields with a 60s ceiling, while the strictly-earlier
health.available gate had 300s. The operator stamps generations.reconciled
only after every process holds its role for minimumUptimeSecondsForBounce
(60s), making reconciled and the health fields that depend on it the last
status fields to converge. The short ceiling closed the gate seconds before
the field landed on a healthy, fully-reconciled cluster, causing intermittent
jsonpath poll timeouts.
Raise those four gates to 300s to match the health.available budget. The waits
stay event-driven; only the ceilings widen.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.
Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add helm-unittest suites for the new Talos templates: MHC maxUnhealthy int-vs-percent rendering, the Talos<->Kubernetes support-matrix guard, the talos templates, and the ported kubelet-reservation math; restore the gpu=on node-label assertions. Remove the obsolete containerd-config-path test and fixtures. Raise tenant control-plane wait timeouts in the e2e runner for Talos PKI + sidecar boot.
Co-authored-by: Andrei Kvapil <kvapss@gmail.com>
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The unmarked CiliumNetworkPolicy fixture carried only an endpointSelector,
which the cilium.io/v2 CRD rejects — spec requires at least one of
ingress/ingressDeny/egress/egressDeny (anyOf), so kubectl apply failed
before the invisible-to-SecurityGroup assertion could run. Add a minimal
ingress rule so the fixture applies as a valid, unmarked policy.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Update the REST storage fixtures and e2e bats to the targetRef shape, and
add cases for the new behaviour: the derived endpointSelector matchLabels,
the apiGroup default to apps.cozystack.io, an empty-targetRef rejection,
and invalid-label-value rejection (over-length or malformed kind/name/
apiGroup). The e2e asserts the dotted-key lineage labels serialise and are
accepted by the real cilium.io/v2 CRD, and that the view reconstructs the
targetRef.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The aggregated API server translates each SecurityGroup into a single
CiliumNetworkPolicy in the same namespace and back. Policies owned by the
SecurityGroup API are marked with the sdn.cozystack.io/securitygroup label,
which the storage always re-asserts on every write and hides from the
SecurityGroup view, so it only ever surfaces or mutates its own policies,
leaves platform-managed policies untouched, and cannot be orphaned by a
tenant overwriting the marker through spec labels. Mutating verbs run the
genericapiserver admission hooks, and updates replace labels/annotations
rather than merging them, matching Kubernetes PUT semantics.
To avoid importing the full Cilium module — which pins a Kubernetes
version incompatible with this project's apimachinery fork — the backing
object is a minimal in-tree CiliumNetworkPolicy mirror that reuses
SecurityGroupSpec as its spec body, making the projection a near-identity
translation in both directions.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Co-authored-by: Timofei Larkin <lllamnyp@gmail.com>
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh)
ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect
can hang indefinitely — observed wedging the E2E install step for ~2h35m after a
test failure until the job-level cancel, with no output after the capture line.
`|| true` only catches a bad exit code, never a hang.
Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s
later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still
beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.
Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.
Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).
Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Timofei Larkin