admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-03 18:47:04 +09:00
Code Issues Packages Projects Releases Wiki Activity
4,293 Commits 544 Branches 294 Tags
release-1.5
Commit Graph
449 Commits
Author SHA1 Message Date
myasnikovdaniil b0d8f3b0ac fix(migrations): pin KubeadmConfigTemplate on the 1.5 slot 45 migration and in the chart (#3892) 
Migration slot 45 holds a different migration here than on main and
release-1.6. On this branch it is the seaweedfs db-split repair, added
after v1.5.3 by `33611dfe7` (the #3370 backport), which in the same
commit bumped `migrations.targetVersion` from 45 to 46. On main and 1.6
slot 45 is the `helm.sh/resource-policy=keep` pin on
`KubeadmConfigTemplate`, and the seaweedfs repair sits at slot 53
instead.

Nothing is broken today. v1.5.0 through v1.5.3 all ship `targetVersion:
45`, so no released cluster is stamped past it and every one of them
still runs 1.6's slot 45 on the way up. Cutting v1.5.4 is what first
ships 46. After that a 1.5.4 to 1.6 upgrade runs `seq 46 53` and never
executes 1.6's slot 45, so helm prunes `KubeadmConfigTemplate` while the
live kubeadm-backed MachineSet still references it,
`bootstrap.configRef` dangles and controller-manager floods through the
talos worker rollover. Tenant kubernetes only, noisy broken upgrade
rather than data loss.

This folds the keep-pin into release-1.5's slot 45 so it does both jobs,
and puts the annotation on the rendered template in
`packages/apps/kubernetes/templates/cluster.yaml` so templates created
after the upgrade are born pinned. `targetVersion` stays 46.

### Why uninstall does not leak

`keep` suppresses only helm's own delete. CAPI stamps an
`ownerReference` to the `Cluster` on this object, which the chart does
not render (checked on a live v1.5 stand, and `MachineDeployment` and
`KubevirtMachineTemplate` carry the same reference). The `Cluster`
itself is helm-managed with no `keep`, so `helm uninstall` deletes it
and GC reclaims the template through that reference.

Template name is deterministic rather than content-hashed, so `keep`
cannot accumulate a copy per upgrade. One residue is the template of a
removed nodeGroup, which helm no longer prunes. It is inert, re-adopted
in place if the group comes back, and GC takes it with the `Cluster`.

### Tests

Mutation-proven rather than revert-tested. The one worth naming:
swapping the selector to the `meta.helm.sh/release-name` annotation
makes the migration exit 0 reporting `pinned=0 failures=0`, which looks
like a clean run, and the test catches it because the annotation never
reached the object.

Also converts four pre-existing negated-grep assertions in
`hack/seaweedfs-guard-parity.bats`. `! grep -q X f` cannot fail, posix
and bash both exempt a `!`-negated pipeline from errexit, so those were
comments that look like assertions. All of them hold once they are real.
 2026-08-19 12:56:34 +05:00
Myasnikov Daniil 314aa5aa84 test(seaweedfs): make the guard-parity discriminator assertions able to fail 
The four assertions in "neither chart classifies on mutable claim
timestamps or liveness" were written as `! grep -qF X "$f"`, which cannot
fail: POSIX and bash both exempt a !-negated pipeline from errexit — "the
-e setting shall be ignored ... if the command's return value is being
inverted with !" — so the command runs, returns 1, and the test carries on
reporting success. Those four were the entire body of the test, so it
asserted nothing whatsoever.

Converting them mechanically turns out to fail, and for a reason worth
keeping: all four names DO appear in both files, in the prose that
explains why each was rejected as a discriminator. `creationTimestamp` and
`readyReplicas` are discussed at length in the comment blocks that record
the decision. So a raw grep would fail on the documentation of the very
invariant the test enforces.

Assert against the template logic instead, with {{/* */}} comment blocks
stripped by a small awk helper. Every comment opener in both files starts
its own line, so dropping whole lines cannot take code with it. What the
test now forbids is a LIVE reference, which is what it always meant.

Proven in both directions: it passes with the explanatory prose in place,
and it fails when a real `readyReplicas` lookup is injected into live
template code in either file.

The awk helper is deliberately a one-liner. cozytest.sh rewrites any bare
`}` in column 0 into `return 0` + `}`, which would corrupt a multi-line
awk body.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-17 15:27:15 +05:00
Myasnikov Daniil 35fb07912e fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls 
Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.

The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.

Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.

The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.

Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.

The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.

The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.

29/29 tests green; sh -n clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit e02adad38f)
 2026-08-17 10:14:35 +00:00
Myasnikov Daniil 11ffa32419 fix(seaweedfs): make naming audit fail closed on kubectl and payload errors 
hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (#3431).

Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.

Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.

Fixes #3431

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 3dc6cbd4f4)
 2026-08-17 10:14:34 +00:00
Myasnikov Daniil f9713b8c98 test(migrations): cover the kubeadm keep-pin folded into slot 45 
Drives the pin end-to-end through the migration script, in the harness the
SeaweedFS half already uses: the real slot 45 executed by path inside the
migrations image's own pinned alpine base, so the interpreter under test
is the busybox ash that run-migrations.sh actually gives it, with only the
cluster boundary faked.

The property asserted is the annotation landing on the object, never a
function being callable. Thirteen cases: an unannotated Helm-managed
KubeadmConfigTemplate gets pinned and the slot stamps 46; several
templates are pinned each in its own namespace; an already-pinned one is
skipped WITHOUT a write; zero matching objects is a clean run; a
CAPI-spawned KubeadmConfig is left alone because it is not Helm-managed; a
cluster with no kubeadm bootstrap provider stamps cleanly; an object that
vanishes between the scan and the pin is skipped rather than fatal; a
failing fleet scan and a failed pin both abort without stamping; a partial
failure still pins the rest and then aborts; both halves run in one pass
with the SeaweedFS hand-over first; a SeaweedFS failure aborts before the
pin is attempted; and migration 43 does not pin, since the two libs must
stay independently sourceable.

The fake kubectl MODELS THE LABEL SELECTOR rather than ignoring it, which
is what makes the characteristic form of this bug detectable. Helm injects
app.kubernetes.io/managed-by=Helm as a LABEL; meta.helm.sh/release-name is
an ANNOTATION and can never be a selector, so using it matches nothing,
the pin silently applies to nothing, and the run reports pinned=0
failures=0 and exits 0. A fake that answered every selector identically
could not tell that apart from a working pin. An ABSENT selector lists
everything, exactly as kubectl does, so dropping the selector shows up as
the pin stamping keep on Machine-owned objects instead of as a quiet
no-op.

Also rewrites the nine pre-existing SeaweedFS "this did not happen"
assertions, which could not fail. POSIX and bash both exempt a !-negated
pipeline from errexit — "the -e setting shall be ignored ... if the
command's return value is being inverted with !" — so `! grep -q X f`
runs, returns 1, and the test carries on to report success. Measured in
this harness, not inferred: with the two halves of slot 45 deliberately
swapped, a `! grep -q 'PIN '` assertion stayed green while the command log
plainly contained the PIN line. The replacement,
[ "$(grep -c X f)" -eq 0 ], puts the result inside `[`, whose non-zero
status does trip errexit, and is the form already used elsewhere in the
file. Their claims turn out to have been true all along, so nothing about
the migrations changes; what changes is that they are now checked — with
the hand-over mutated to re-annotate a Cluster that already carries keep,
"idempotent: a Cluster already owned by <name>-db AND carrying keep is
left alone" now goes red where before it passed.

Every case was confirmed against a deliberately broken pin before being
made to pass: annotation-as-selector, no selector at all, best-effort
instead of fail-closed, an unconditional write instead of read-then-skip,
a vanished object treated as fatal, the halves swapped, and the pin never
called.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-17 14:58:00 +05:00
myasnikovdaniil 22414e604f fix(postgres-operator): align CNPG CRDs with 1.27.3 (#3526) 
## What this PR does

Aligns the vendored CloudNativePG chart metadata and CRDs with the
already-pinned 1.27.3 operator image. CNPG 1.27.3 stores the
instance-manager session identifier in
`Backup.status.instanceID.sessionID`; the 1.27.1 CRD prunes that field,
causing false instance-manager restart errors and failed backups.

Fixes #3479.

### Screenshots

Not applicable.

### Release note

```release-note
fix(postgres-operator): align CloudNativePG CRDs with the pinned 1.27.3 operator image so PostgreSQL backups no longer fail with false instance-manager restart errors
```
 2026-08-12 19:00:11 +05:00
myasnikovdaniil b7f2803e84 [Backport release-1.5] fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover (#3532) 
## What this PR does

Hand-backport of #3034 to `release-1.5`. The `backport` label was
applied at merge time but no backport PR was ever opened, so the change
never reached this branch — found by auditing the branch with
`cmd/backport-audit`, which reported it as `MISSING`.

Every candidate that went missing this way merged before #3155 landed,
when `conflict_resolution` was passed as a top-level input instead of
nested under `experimental`. The action ignored it and fell back to
`fail`, which on a conflicting cherry-pick opens no PR and reports no
failure, so these were dropped with no draft to find and no red check to
notice.

The six original commits are cherry-picked with `-x`, unchanged and in
order. No adaptation was needed: the cherry-pick was clean, and the net
diff is identical to the change on `main`.

### Verification

The two `.diff` files are image patches applied at build time against
upstream `container-object-storage-interface`, so what matters is that
this branch pins the same upstream source. It does — both `release-1.5`
and `main` set `ARG COMMIT_REF=v0.2.2` in
`images/objectstorage/Dockerfile`, and the resulting patch set on this
branch (`91`, `92`, `93`) is byte-identical to `main`'s. Patch `91` was
already present here unchanged, so `92` applies over the same base it
does on `main`.

The branch differs from `main` only in its builder pins (`golang:1.24`
versus `1.26`, unpinned base digests), none of which affect patch
application.

### Screenshots

Not applicable — no UI change.

### Release note

```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
 2026-08-04 14:48:50 +05:00
myasnikovdaniil be9094ce95 [Backport release-1.5] fix(seaweedfs): close the 4.31 rename fallout on the 1.5.x→1.6 upgrade path (#3370) 
# Description
Backport of #3339 to `release-1.5`.
 2026-08-04 13:30:28 +05:00
Aleksei Sviridkin 85dc51c48a test(e2e): assert bucketaccess accessGranted=true and dump controller pods on bucket failure 
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 0855ff2d1b)
 2026-08-04 13:18:15 +05:00
Aleksei Sviridkin 400856250d test(bucket,harbor): assert BucketClaim converges to bucketReady promptly 
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.

Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 63d14e7487)
 2026-08-04 13:18:15 +05:00
Myasnikov Daniil f716e2f81d fix(postgres-operator): align CNPG CRDs with 1.27.3 
The operator image is pinned to 1.27.3 while the vendored chart and
CRDs still advertise 1.27.1. Kubernetes prunes Backup
status.instanceID.sessionID, causing false instance-manager restart
failures.

Import the official 1.27.3 CRDs, align the chart appVersion, and preserve
the overlay across make update. Add render and API round-trip regression
coverage.

Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-04 12:07:29 +05:00
myasnikovdaniil d2cf83ccc9 fix(mariadb): widen startup probe budget so bootstrap cannot be killed 
Backport of #3344 onto release-1.5.

The bot's cherry-pick imported hack/e2e-chainsaw/mariadb/chainsaw-test.yaml
whole, because release-1.5 has no such file: the app suite here is still
BATS, and the Chainsaw port landed with #2826 on main only. That file is
dropped and its widening applied to the gate that actually runs on this
branch instead.

That gate needed the change more than the chainsaw one did. In
hack/e2e-apps/mariadb.bats both tests wait 80s for an endpoint address to
appear, and an address appears only once a replica has cleared its startup
then readiness probe. This commit raises the startup budget to 310s, so an
80s ceiling would fail runs the probe was still willing to wait for --
importing the chainsaw file and leaving the BATS suite alone would have
turned the fix into an e2e failure. Both waits go to 600s, matching the
10m the upstream chainsaw assert uses and for the same reason.

packages/apps/mariadb/Makefile gains the `test:` target. It is absent on
release-1.5 -- it arrived with #3071, which was not backported -- and
hack/helm-unit-tests.sh discovers suites by probing for that target, so
without it the backported tests/startup_probe_test.yaml would never be
executed by `make unit-tests`.

Verified locally: 4 assertions in tests/startup_probe_test.yaml pass.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit a37715062e5da24e54956f075dd2d6417c4dff2e)
 2026-07-31 10:18:04 +05:00
Myasnikov Daniil 33611dfe7e fix(seaweedfs): close the 4.31 rename fallout on the 1.5.x upgrade path 
Backport of #3339 onto release-1.5, on top of the #3282 adopt-in-place
fix (#3326).

Three differences from main, all forced by what release-1.5 does not
carry:

  * The repair migration is renumbered 53 -> 45 and stamps 46, since
    release-1.5 tops out at migration 44 with targetVersion 45.
    platform values.yaml goes to targetVersion 46.

  * Migrations 43 and 45 keep release-1.5's inline labelled stamp
    instead of lib/cozystack-version.sh. That helper is the shared
    stamp refactor (#2980), whose backport (#3065) was closed, so it
    does not exist on this branch. Only lib/seaweedfs-db-adopt.sh --
    the part that actually carries the fix -- is sourced.

  * The extra/seaweedfs post-delete cleanup hook and its three tests
    are dropped. The hook arrived with #3092, a breaking change that
    was never backported, so there is nothing here for #3339's changes
    to it to apply to. Comments in lib/seaweedfs-db-adopt.sh and the
    recovery runbook that promised the hook would reclaim an orphaned
    volume are corrected to say it does not on 1.5.x.

The migration renumbering is carried through the shared helper, the
bats suite (STAMP 54 -> 46) and the runbook prose.

Verified locally: helm unittest green for both seaweedfs charts (51 + 16
assertions) including both blind-upgrade guard canaries, and 34 bats
assertions across migration-seaweedfs-db-adopt, seaweedfs-naming-audit
and seaweedfs-guard-parity.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 88e9fdcbbbff8aeab317b08557ca6bd6c8b5706d)
 2026-07-31 10:18:01 +05:00
Алексей Артамонов e115e62fb1 test(mariadb): fix operator MariaDB CR name in single-replica failure dump 
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
(cherry picked from commit f8e4222b7d)
 2026-07-02 19:54:31 +00:00
Алексей Артамонов b5a09cd14b fix(mariadb): re-guard replication so single-replica passes the operator webhook 
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.

Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:

- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
  external access via `service.type: LoadBalancer` and keep `primaryService`
  LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
  (no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
  -primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
  installs are visible in the dashboard.

Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
(cherry picked from commit c7e8f4d975)
 2026-07-02 19:54:31 +00:00
Myasnikov DaniilandClaude Opus 4.8 5653c30799 fix(e2e): bound per-test crust-gather snapshots with a timeout 
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh)
ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect
can hang indefinitely — observed wedging the E2E install step for ~2h35m after a
test failure until the job-level cancel, with no output after the capture line.
`|| true` only catches a bad exit code, never a hang.

Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s
later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still
beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-20 20:48:51 +05:00
Myasnikov DaniilandClaude Opus 4.8 6deddc6011 fix(e2e): refuse cilium endpoint disconnect for any non-terminal owner 
The leak-healer only refused to disconnect an endpoint whose owning pod was
Running, so a Pending owner that already held a live IPAM allocation
(status.podIP is populated once the sandbox is created, before the pod goes
Running) could have its endpoint evicted mid-startup. Refuse for any
non-terminal phase (!= Succeeded && != Failed); only a terminal owner is
safely past needing its endpoint.

Addresses non-blocking follow-up 4 on #2948.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-20 16:35:39 +05:00
Myasnikov Daniil 9beda4e60e fix(e2e): reach tenant API via LoadBalancer instead of port-forward 
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.

Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.

Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).

Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-20 11:23:05 +05:00
Andrei KvapilandClaude cc9923061f fix(e2e): grant the cilium leak-healer delete on pods 
The healer's reserved/infra-endpoint remedy and its disconnect-failure
fallback both run `kubectl delete pod` to reschedule the wedged pod onto a
free IP, but the ClusterRole granted pods only get/list — so the delete was
forbidden and the pod stayed wedged. Add delete to the pods rule.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-06-19 21:30:38 +02:00
Myasnikov Daniil 51496832e7 fix(e2e): capture tenant cluster snapshot from an in-subshell trap 
crust-gather can only reach a tenant Kamaji cluster through the test's
localhost port-forward (it connects directly to the kubeconfig server URL —
there is no apiserver-proxy mode — and the in-cluster URL is unreachable from
the runner). cozytest.sh's generic EXIT trap runs in the parent shell AFTER
the failing test subshell has exited and its port-forward is reaped, so a
collect there hits a dead socket and crust-gather bails before writing
anything (every tenant snapshot so far came back empty).

Capture the tenant from an EXIT trap registered INSIDE run_kubernetes_test
instead: it fires during THAT subshell's exit, while the supervised
port-forward is still alive. Verified end-to-end against the real
hack/cozytest.sh under dash (CI's /bin/sh) in an ubuntu:22.04 container —
the in-subshell trap fires on failure with the PF alive, the tenant snapshot
lands in the same snapshots/<test>/ dir as the host snapshot, the generic
cozy_cleanup still runs, and there is no pipeline deadlock.

Drop the now-dead tenant loop from cozytest.sh's generic trap (it could only
ever see a reaped PF); it keeps the host snapshot.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-19 22:30:41 +05:00
Myasnikov Daniil 391c27f11e fix(e2e): supervise tenant API port-forward so a drop self-heals 
A single `kubectl port-forward` to the tenant Kamaji API silently dies on
an SPDY/connection drop and never reconnects. The tenant apiserver itself
stays healthy (control-plane pods observed at 0 restarts while localhost
refused connections), so this is a client-side port-forward death, not a
control-plane outage. Once it dies every later tenant kubectl op wedges:
a long `kubectl wait` burns its full timeout against the dead socket
(observed: nfs-test-pod hit its 10m deadline) and the failure-path
`describe`/`get events` print 'connection refused', destroying the
diagnostics too.

Run the port-forward under a respawn loop so a drop self-heals within ~1s;
`kubectl wait` retries across the gap and the in-test failure dumps work
again. Track the supervisor PID in a pidfile so cozy_cleanup() can stop it
on a failed run (the success path kills it directly), and reap the child
port-forward with pkill in both paths.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-19 18:36:03 +05:00
Andrei Kvapil 770ce36922 fix(e2e): heal cilium IP conflict held by a reserved endpoint 
The endpoint-leak healer evicts the stale endpoint holding a disputed IP
via 'cilium-dbg endpoint disconnect'. But when the IP is held by a
reserved/infra endpoint with no k8s pod identity — e.g. reserved:ingress,
the per-node Cilium Ingress endpoint — that call is rejected with
'endpoint may not be associated reserved labels', so the healer spins
forever and the wedged pod (observed: cozy-backup-controller) never
recovers, failing the run.

Such infra endpoints legitimately own the IP; the duplicate is an IPAM
race that also handed it to a pod. Evicting infra is both impossible and
wrong. Instead reschedule the wedged pod onto a free IP by deleting it
(verified live: the recreated pod lands on a clean address while the
ingress endpoint keeps the disputed one). Also fall back to deleting the
wedged pod if a disconnect of a presumed pod endpoint fails for any
reason, so a pod is never left wedged.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 19a9063ae0)
 2026-06-19 13:15:43 +05:00
Andrei Kvapil d7f244fcf6 test(e2e): give harbor BucketClaim readiness a 10m budget 
Harbor's registry BucketClaim wait timed out at 300s on an overloaded
runner even though the underlying Bucket reached bucketReady=true and the
BucketAccess was granted — only the central COSI controller's propagation
of readiness back onto the namespaced BucketClaim lagged (all HelmReleases,
harbor included, were Ready by end-of-run). Raise the wait to 600s to match
the dependent harbor-system HelmRelease budget and remove the flake.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 074501f999)
 2026-06-19 13:15:43 +05:00
Myasnikov DaniilandClaude Opus 4.8 50e5f94132 test(e2e): per-test crust-gather snapshots + cleanup-on-exit via cozytest.sh 
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.

run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.

Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 18:07:52 +05:00
Myasnikov DaniilandClaude Opus 4.8 bf96e61a53 fix(e2e): deploy cilium-leak-healer as a real @test, not a setup_file hook 
The e2e runner hack/cozytest.sh converts only @test blocks into functions and runs those; it never invokes bats setup_file/teardown_file. So the healer's setup_file deploy silently never executed (confirmed: the Job was absent from the cozyreport while the same manifest deploys and runs fine on a real Talos cozystack cluster). Move the deploy into a real, first @test so cozytest.sh runs it (with visible set -x trace) before the install churn, and assert the Job landed without ever failing the suite (band-aid).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 15:02:35 +05:00
Myasnikov DaniilandClaude Opus 4.8 9e43d455f0 test(e2e): wait for tenant nodes Ready on a single deadline, not split budgets 
kubernetes-latest flaked because the worker-readiness wait was split into 8m-to-join + 3m-to-become-Ready. Under sandbox load a slow KubeVirt VM boot consumes most of the join budget, then the tenant cluster's cilium CNI needs several more minutes to make the freshly-joined nodes Ready, overflowing the fixed 3m Ready window even though the CNI converges fine (its HelmRelease reached True shortly after). Replace with one 12m deadline polling for >=2 Ready nodes, robust to wherever the time goes. Same single-deadline shape as the LINSTOR post-install fix (#2928).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 15:02:34 +05:00
Myasnikov DaniilandClaude Opus 4.8 763b85eb28 test(e2e): wait for EtcdBackupSchedule instead of a bare get in etcd.bats 
Reconfiguring the Etcd CR triggers a HelmRelease upgrade that renders the EtcdBackupSchedule; hr/etcd and etcdcluster can report ready on the prior revision before the new resource materializes, so the immediate get raced to NotFound. Poll for it (timeout 120s), mirroring the CronJob wait already used later in the same test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 14:08:48 +05:00
Myasnikov DaniilandClaude Opus 4.8 c1aca89fde fix(e2e): drive LINSTOR post-install waits off a single 15m deadline (#2928) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 10:40:06 +05:00
Myasnikov DaniilandClaude Opus 4.8 d3958c5bcb refactor(build): mode=max registry cache with a main-only warmer (#2938) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 10:40:05 +05:00
Myasnikov DaniilandClaude Opus 4.8 9ce0611b30 test(e2e): self-heal Cilium orphaned-endpoint leak across install and apps (#2874) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 10:40:05 +05:00
Myasnikov DaniilandClaude Opus 4.8 40431de0b5 test(e2e): fix bucket.bats port-forward and S3 client reliability (#2944) 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-18 10:40:05 +05:00
Myasnikov DaniilandClaude Opus 4.8 96b9f5eac6 test(e2e): pin Kubernetes to v1.33.12 to avoid KCM VAP type-checker panic 
The e2e cluster bootstrap let `talosctl gen config` pick its bundled
default Kubernetes version (v1.33.1 for Talos v1.13.0). That version
predates the fix for the kube-controller-manager ValidatingAdmissionPolicy
status type-checker nil-pointer panic on `additionalProperties: true`
schemas (kubernetes/kubernetes#135155, backported to release-1.33 in
#136958, first released in v1.33.10). cozystack-api's aggregated Tenant
schema publishes `additionalProperties: true`, so on the older default
KCM crash-loops cluster-wide during install and fails E2E.

Pin the e2e management cluster to v1.33.12 (latest 1.33 patch) so it runs
a Kubernetes that contains the fix. This complements the schema-side fix
in #2867 and is independent of it.

Refs: #2863

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-10 20:22:07 +03:00
Aleksei Sviridkin b6108039b7 feat(gpu-operator): add container variant for preinstalled host driver (#2766) 
## What this PR does

Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).

The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.

The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:

- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.

Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).

The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.

Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.

Closes #2764

### Release note

```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.

* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
  * Clarified Talos native pod example wording.

* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-10 02:00:59 +03:00
Aleksei Sviridkin fb070ffa21 feat(hack): rewrite check-readiness as a Go command with test suite (#2755) 
## What this PR does

Replaces the `hack/check-readiness.sh` diagnostic with a Go command at
`cmd/check-readiness`, integrated into the build the same way as
`cozypkg`:

- `make check-readiness` builds `_out/bin/check-readiness`, with
`Version`
  injected via `-ldflags`.
- The tool shells out to `kubectl` and reports only **non-ready**
cozystack /
  Flux / Kubernetes resources, parsed from status conditions (not column
heuristics). Flags: `--wait`/`--timeout`, `--watch`, `--core`,
`--parallel`,
  `--verbose`, `-n`/`-l` scoping, `--no-color`, `--version`.

Adds a black-box golden test suite under `test/check-readiness/`:

- A mock `kubectl` serves canned fixtures; the suite diffs
stdout/stderr/exit
against golden files across 30 scenarios (arg parsing,
condition/phase/suspend
handling, CRD-not-installed, kubectl arg propagation, wait/timeout, edge
  cases) plus a PTY test for color output.
- `make test-check-readiness` runs it; wired into `make unit-tests`.

Supersedes #2740, which added the same feature set to the bash script.

### Release note
```release-note
feat(hack): replace check-readiness.sh with a `check-readiness` Go command (build with `make check-readiness`); reports non-ready cozystack/Flux/Kubernetes resources with --wait/--watch/--core/--parallel/--verbose and namespace/label scoping.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Added `check-readiness` command for verifying Kubernetes resource
readiness status across namespaces
* Supports multiple execution modes: one-shot, wait with timeout, and
watch with configurable intervals
* Includes namespace filtering, label selectors, parallel resource
fetching, and color output control

* **Tests**
* Added comprehensive test suite validating command behavior across
various scenarios

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2755?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-09 23:52:12 +03:00
Aleksei Sviridkin c961ffe286 feat(ci): test-impact analysis E2E (default-on) + release E2E workflow (#2559) 
## What this PR does

Adds **test-impact analysis (TIA)** to the PR E2E workflow so that only
bats files affected by the diff are exercised, and a **release-time E2E
workflow** that runs the full suite on every release tag to keep
coverage on the shipped code.

### 1. Test-impact analysis selector — `hack/select-e2e.sh`

Reads `packages/core/platform/sources/*.yaml` (the PackageSource
dependency graph) and emits the bats files affected by a PR diff. The
walk is the same one `cozypkg dot` renders.

| Input | Output |
|---|---|
| `packages/apps/postgres/values.yaml` | `postgres` |
| `packages/system/postgres-operator/...` | `postgres harbor`
(transitive) |
| `packages/system/cilium/...` | full suite (cilium has many transitive
`*-application` dependents) |
| `packages/library/cozy-lib/...` | full suite (library affects
everything) |
| `hack/e2e-install-cozystack.bats` | full suite (shared install affects
all apps) |
| `hack/e2e-apps/redis.bats` | `redis` (per-app edit, never escalates) |
| `docs/agents/contributing.md` | (empty — no E2E impact) |
| `dashboards/gpu/gpu-fleet.json` | (empty) |

Conservative fallbacks:
- An unrecognised `packages/*` path or a system source with no
`*-application` descendants escalates to the full suite, so a path
inside the graph is never silently dropped.
- Per-app bats edits are matched **before** the full-suite trigger, so
editing one bats file selects only that app rather than escalating.

### 2. Default-on TIA + `full-e2e` opt-out — `pull-requests.yaml`

- The `Select E2E tests` step runs by default and produces `apps` +
`skip` outputs.
- `Run E2E tests` runs unless `skip=true` (docs / dashboards / `*.md`
only).
- Adding the `full-e2e` label skips the selector step (output empty) and
falls through to the unchanged full-bats-list path — the safety hatch
when reviewers want the full suite on a PR.

### 3. `release-e2e.yaml` — full E2E on every release tag

Triggered on `v*.*.*`, `v*.*.*-rc.*`, `v*.*.*-beta.*`, `v*.*.*-alpha.*`.
Builds images + Talos image, then runs the full suite. Independent of
`tags.yaml`'s release publish flow — failure here is an alarm, not a
release blocker.

This closes the coverage gap left by making PR runs default to TIA: the
full suite is now exercised at tag-cut time against the shipped code.

## Tests

`hack/select-e2e_test.bats` covers 11 cases — single app, transitive
operator, networking/library full-suite, docs/dashboards skip,
`kubernetes-application` two-bats mapping, shared E2E helper, install
bats, per-app bats no-escalation, release-e2e workflow change.

YAML parse verified for both new workflow files.

## Workflow injection safety

All `${{ }}` contexts that could carry untrusted data
(`github.base_ref`, label arrays) flow through env vars, never directly
into shell text. `github.workspace` is also indirected through
`WORKSPACE` env in the release workflow.

### Release note

```
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* CI now runs a Test Impact Analysis to pick only affected apps,
reducing E2E runtime.
* Added a release-level E2E workflow that runs full acceptance tests for
version tags with deterministic sandboxing, artifact capture,
conditional retries, teardown, and report upload.
* E2E jobs accept selected-apps metadata and support conditional
full-suite overrides.

* **Tests**
* New test suite validates selection logic, operator expansion,
full-suite escalation, and docs-only/no-op behaviors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-09 23:51:24 +03:00
Aleksei Sviridkin 479929ebfb ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown (#2558) 
## What this PR does

Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.

On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.

> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (7b9f286fa), making a blanket 2×
redundant.

**Fixes gateway.bats teardown leakage.** The nested-tenant tests deleted
tenants fire-and-forget, parent and child back-to-back. The leftover
uninstalls (each blocked on a cleanup Job, parents wedged on
still-terminating child namespaces) plus one mid-install child HR
occupied exactly 5 workers on the `--concurrent=5` tenants
helm-controller shard, starving whichever app test ran next — observed
as the harbor HR sitting unreconciled for its whole 5m HR-Ready budget
in [run
27020081550](https://github.com/cozystack/cozystack/actions/runs/27020081550),
surfaced by this PR's own retry removal + diagnostics dump. Teardown now
deletes child→parent with hard `wait hr --for=delete` between, so a
wedged tenant uninstall fails gateway.bats itself, not an innocent
neighbor.

## Why

Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry loop
never recovered a flake, only stretched deterministic failures and
tripled diagnostic wall-time. Same data shape on `Install Cozystack`.

Beyond wasted CI time, the retry was hiding ~10 deterministic bugs (Helm
namespace-ownership conflict, seaweedfs HR timeout, harbor BucketInfo
wiring, vminstance disk race, etc.). Each failure looked like a "flake"
because the retry sometimes coincided with whatever transient state had
cleared — the retry never fixed the bug, just delayed surfacing.

## Dependencies

The deterministic bugs the retry was masking are now fixed on `main`:
- ✅ **#2508** — installer namespace bootstrap (Helm namespace-ownership
conflict on cold install) — merged
- ✅ **#2509** — operator HelmRelease config knobs (`seaweedfs-system`
2-min wait race within Flux's 5-min reconcile windows) — merged
- ✅ **#2528** — harbor bucket-secret + BucketInfo gating (harbor
`ValuesError` on first install) — merged
- ✅ **#2529** — objectstorage-controller BucketAccess conflict retry —
merged

Companion PRs in the #2619 split (independent of this PR,
ordering-wise):
- **#2602** — Flux v2.8.0 + chart fixes
- **#2601** — seaweedfs-system split

This PR does NOT depend on #2602/#2601 — it now touches only the
workflow file and gateway.bats teardown, both on top of fresh `main`.

Surfaced from #2500.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* CI prepare-environment step now reports plain attempt counts with
clear success/failure messages.
* Install and per-app test steps no longer retry; each runs once and
fails immediately on error. Failed apps log diagnostics and job proceeds
to remaining apps while overall job fails.
* **Tests**
* End-to-end tests and install/prepare flows use longer, more tolerant
timeouts and added existence polling to reduce flakiness and improve
diagnostics.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2558?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-09 16:21:15 +03:00
Myasnikov Daniil 9868c43e53 fix(build): read --cache-from from ghcr :latest so PR builds warm-start 
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.

Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-09 14:24:36 +05:00
Aleksei Sviridkin 9352d3ee0e feat(gpu-operator): add container variant for preinstalled host driver 
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.

The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.

- driver.enabled=false        use the host driver
- toolkit.enabled=false       host already has nvidia-container-toolkit;
                              enabling overwrites the containerd
                              config.toml the host configured via
                              nvidia-ctk runtime configure
- devicePlugin.enabled=true   publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false   do not unbind the host driver
- cdi.enabled=false           CDI spec generation is serviced by the
                              disabled toolkit; an apt host uses the
                              legacy nvidia-container-runtime hook, so
                              pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false   regular pod workloads, not VMs

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-08 16:58:51 +03:00
Myasnikov Daniil 53edfc1f50 test(e2e): capture COSI bucket state in failure diagnostics and cozyreport 
Run 27057500380 failed on harbor's BucketClaim never reaching
bucketReady=true, and neither artifact could explain why: the per-app
diagnostics dump has no bucket objects, and cozyreport collects
neither COSI resources nor objectstorage-controller / seaweedfs
provisioner logs — by the time the report is taken, the bucket state
has moved on.

Add a COSI module to cozyreport.sh (bucketclaims/bucketaccesses/
buckets/classes as custom-column tables plus full YAML, controller and
per-namespace provisioner logs) and a bucket-readiness snapshot to the
per-app failure dump in the PR workflow. The COSI CRDs ship no printer
columns, so plain kubectl get would show only NAME/AGE — readiness
fields are pulled explicitly.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-07 18:10:42 +05:00
Myasnikov Daniil 5b986e0a47 test(e2e): wait out nested-tenant teardown in gateway.bats 
The last three gateway tests create nested tenants (gwprop,
gwparent/gwchild, rparent/rchild) and deleted them fire-and-forget,
with parent and child deletes back-to-back. The file exits in ~25s
while the uninstalls run on for minutes: each tenant uninstall blocks
on a cleanup Job in cozy-system, and a parent deleted alongside its
child wedges on the child's still-terminating namespace.

On the tenants helm-controller shard (--concurrent=5) the leftovers
from one gateway.bats run — three stuck parent uninstalls, one stuck
child uninstall, and the rchild HR still mid-install — occupied
exactly 5 workers, so the next test in line (harbor) sat unreconciled
for its whole 5m HR-Ready budget and failed without ever erroring.

Sequence the teardown instead: delete child, wait for its HR to go
away, then delete parent and wait again. The waits are hard failures
on purpose — if a tenant uninstall genuinely wedges, gateway.bats is
the test that should go red, not whichever app test runs next.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-06 13:17:31 +05:00
Myasnikov Daniil 67b1a960dd test(e2e): fail install test when HelmReleases are not Ready 
The HR readiness gate in e2e-install-cozystack.bats was toothless in two
layers: the backgrounded kubectl-wait fan-out discarded child exit codes
(POSIX `wait` without args returns 0), and the final check echoed
"Some HelmReleases failed to reconcile" without exiting non-zero — a
regression from 1f240387f, which replaced the unavailable bats `fail`
helper with a bare echo. A permanently-failing platform HelmRelease
(e.g. the backup-controller CRD rejection, #2822) shipped through green
CI for weeks.

Replace the fan-out with a single `kubectl wait hr --all -A` for trace
visibility, gate on an outcome-based re-list (which also covers HRs
created after the snapshot, with a 120s retry to absorb momentary
drift-reconcile flaps), and dump the full Ready condition message per
non-ready HR so the real error is visible in the test output.

Refs #2822

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-04 15:19:28 +03:00
Aleksei Sviridkin 2363f2d9be fix(backups): carry dropdown option sources in CRD annotations, not schema 
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.

Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.

A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-04 12:45:14 +03:00
Andrei KvapilandClaude 4dc2367fe4 test(tests): exclude loop devices from host LVM scanning 
Add an /etc/lvm/lvm.conf overwrite to the e2e Talos machine config so the
host LVM ignores /dev/loop devices (alongside drbd/dm/zd). This prevents the
host from scanning and activating volume groups inside loop-mounted disk
images used during e2e.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-06-03 14:00:16 +03:00
Aleksei Sviridkin 4c34e0f427 build(codegen): reproduce x-cozystack-options on backup CRDs after controller-gen 
controller-gen cannot emit the x-cozystack-options vendor extension, so the
annotations on the backup CRDs (backupjobs/backups/plans/restorejobs) were
hand-edited into generated output — the next make generate would strip them,
breaking the appkind/plan/backup form dropdowns and failing the codegen-drift
check.

Add a deterministic awk pass that re-applies the extension after
controller-gen, driven by per-CRD field targets. It inserts lines without
reserializing the YAML, so it reproduces the committed CRDs byte-for-byte and
is independent of the host yq version.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-02 17:38:27 +03:00
myasnikovdaniil c435efa199 feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602) 
## What this PR does

Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.

Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).

Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.

## Commits

**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).

**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.

**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).

**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.

## Scope discipline

This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.

## Verification

- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.

### Release note

```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).

When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
  * New SeaweedFS DB Helm chart and optional managed DB release.

* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.

* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
  * Removed aggressive HelmRelease force-upgrade/install flags.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-02 14:08:39 +05:00
Aleksei Sviridkin 2d58479f06 feat(platform): add kuberture as optional system package (#2647) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional
system package. Kuberture is a small controller that translates the
`default/kubernetes` EndpointSlice into headless Services with
external-dns-style annotations, closing the gap that external-dns cannot
use EndpointSlice as a source directly:

```text
EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS
```

The package is opt-in via `bundles.enabledPackages:
[cozystack.kuberture]`. Operators populate `config.outputs` per
HelmRelease — each output targets a specific external-dns instance by
its `annotationPrefix`, so a single kuberture install can serve multiple
external-dns instances (e.g. one publishing to the public zone, another
to an internal corporate prefix) simultaneously.

The vendored chart at `packages/system/kuberture/charts/kuberture/` is
pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and
pinned by both version AND OCI manifest digest in the Makefile
(mirroring the ouroboros lockstep pattern). The controller image at
`ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim
values.yaml. Both pins advance in lockstep with each upstream release;
the procedure is documented inline.

Install-order dependencies declared in `PackageSource`:
`cozystack.networking` (CNI before pod scheduling) and
`cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the
chart's enabled-by-default ServiceMonitor manifest). External-dns is
deliberately NOT a dependency — kuberture creates annotated Services
regardless of whether external-dns is installed; external-dns is a
downstream consumer, not a runtime prerequisite.

Coverage:

- **helm-unittest**
(`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases):
fail-fast on empty outputs, ServiceMonitor enabled by default, source
pinned to `default/kubernetes`, multi-output prefix isolation,
empty-string `annotationPrefix` schema rejection, omitted-prefix
fallback to controller default, ConfigMap → Deployment volume mount,
controller image pinned by tag+sha256 digest, Role permits full Service
lifecycle, metrics containerPort derived from `metricsBindAddress`.
- **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package
cozystack.kuberture` with two outputs (different `annotationPrefix`
values), waits for HelmRelease and controller readiness, asserts both
output Services are headless and carry the right
`hostname`/`target`/`ttl` annotations under their configured prefix,
cross-checks per-output isolation (each Service only carries its own
prefix's annotations).

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not applicable — no UI changes.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Adds kuberture as an optional system package to publish the Kubernetes
API endpoint to DNS via configurable external-dns outputs; includes Helm
chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart
values/schema.

* **Tests**
* Adds Helm unit tests and an end-to-end smoke test validating
controller readiness, headless Service annotations, and external-dns
routing/probe behavior.

* **Documentation**
* Adds a full README with usage, configuration examples, and
vendoring/pinning guidance.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---

Documentation companion:
[cozystack/website#539](https://github.com/cozystack/website/pull/539).
 2026-06-02 09:40:26 +03:00
Arsolitt 17b2fb70a3 feat(kafka): add TLS support via Strimzi listener configuration (#2681) 
## What this PR does

Adds TLS support to the Kafka managed service via Strimzi listener
configuration.

- Introduces a `tls.enabled` field that controls the external listener
(port 9094). When unset, defaults to the value of `external` (auto-on
when external access is enabled). Explicit `tls.enabled` always wins.
- Strimzi manages the cluster PKI automatically; the operator publishes
`<release>-cluster-ca-cert` and `<release>-clients-ca-cert` for client
trust setup. No cert-manager chain is rendered by this chart — the
operator owns the chain end-to-end.
- The internal TLS listener on 9093 is always on (Strimzi default); this
field only governs the external listener on 9094.
- External LoadBalancer listener is decoupled from `tls.enabled`: it is
gated on `external: true` only. Previously, an explicit `tls.enabled:
true` with `external: false` would force the external listener to be
rendered.

Verified end-to-end on a sandbox cluster: external listener exposes a
TLS endpoint signed by the Strimzi-managed cluster CA, client
connections with the published trust bundle succeed.

Note: this chart does not configure listener authentication. With TLS on
the external listener, clients can connect without credentials unless
`KafkaUser` resources with `tls`/`scram-sha-512`/`oauth` authentication
are configured outside this chart.

### Release note

```release-note
feat(kafka): add TLS support on the external listener via Strimzi (tri-state tls.enabled)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tri-state TLS for the external Kafka listener: explicit on, explicit
off, or inherit from the external toggle; chart values now include a
top-level tls object.

* **Documentation**
* Expanded chart docs/README clarifying tls.enabled inheritance,
plaintext risk when disabled with external access, and authentication
guidance.

* **Tests**
* Added/updated schema, rendering, RBAC/secret-naming, helm unit, and
end-to-end tests covering TLS tri-state and external-listener behavior.

* **Bug Fixes**
* RBAC and secret-name expectations updated to use the `-cert` suffixed
secret names.

* **Chores**
  * Added a Makefile test target to run helm unit tests.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2681?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-29 13:12:42 +03:00
Myasnikov Daniil a3a7a6d516 test(e2e): add HR existence backstop for external-dns ${name}-system 
The two external-dns bats tests already poll for the user-facing ${name}
HelmRelease before calling kubectl wait, but they immediately wait on the
${name}-system HR without the same backstop. The system HR is materialised
asynchronously by the controller, so kubectl wait can race and exit non-zero
when the object does not exist yet. Apply the same existence poll pattern
to both occurrences.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-28 21:25:25 +05:00
Myasnikov Daniil 401993ac62 test(e2e): wait for parent HR Ready before downstream asserts 
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape:
the bats steps assumed a side-effect ordering ("by the time X is true,
the parent HR's helm install is done") that held under v2.7's faster
dispatch but breaks under v2.8 where helm-controller walks kstatus on
every applied resource before flipping the HR Ready.

run-kubernetes.sh:
  After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx)
  reaches Ready, the remediation guard reads `.status.history` on the
  parent kubernetes-${test_name} HR. The parent's helm install can still
  be Reconciling at that point because kstatus is finishing its full
  sweep; `.status.history` is empty until install completes, so the
  guard exits 1 with "empty .status.history on a Ready HR". kubernetes-
  latest masked this because the ouroboros assertions add ~10m of
  buffer; kubernetes-previous (no ouroboros) hit the guard with the
  parent still installing. Add an explicit
    kubectl wait hr/kubernetes-${test_name} --for=condition=ready
  before the guard.

vminstance.bats:
  The test waited 120s for `kubectl get vmi vm-instance-$name` to
  return a VMI with an assigned IP, then waited for the parent HR Ready
  separately afterwards. Under v2.7 the parent HR's helm install
  applied the VirtualMachine CR fast enough that kubevirt had created
  the VMI before the 120s budget expired. Under v2.8 the dispatch is
  slower and the VMI poll returned NotFound for its entire budget.
  Reorder: wait for HR Ready first (which implies the VirtualMachine
  CR is applied + kstatus-ready), then poll for the VMI with IP.
  Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c664b868bc)
 2026-05-28 12:43:50 +05:00
Myasnikov Daniil 7b9f286faa test(e2e): bump app HR-Ready waits for Flux v2.8 kstatus 
Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.

Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.

Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.

Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.

Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5ef02ee7b3)
 2026-05-28 12:43:50 +05:00