## What this PR does
Implements the immutable-tag + rc-promotion flow from #2677. A **stable
release becomes a renamed release-candidate**: the bytes shipped as
`vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as
`vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt —
it is *promoted* by retagging the rc's existing images.
Layered onto the build matrix from #2937/#2983 (this PR is **stacked on
`refactor/build-matrix-2937`** and must merge after it).
## The five force-retag sites, removed
| Site | Before | After |
|---|---|---|
| `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` |
write-once (create-if-absent, fail if it would move) |
| `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` |
compare-before-force (no-op if unchanged; staging branch only) |
| `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` |
write-once at the PR merge commit (force impossible by construction) |
| `pull-requests-release.yaml` maintenance branch | `updateRef
force:true` | fast-forward-only |
| `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow
deleted** — stable only via explicit promote |
## Version decoupling (the enabler)
The operator baked its version into the image at build time, so an rc
image self-reported the rc string — blocking retag-promotion. It now
reads `COZYSTACK_VERSION` from the environment (threaded via
`cozystackOperator.platformVersion` → Deployment env, stamped by `make
manifests`), falling back to the build-time value when unset. The same
image bits can report any release name. The only runtime reader is the
telemetry metric `cozy_cluster_info{cozystack_version=...}`.
## Promotion flow
`promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`):
1. Validate the rc release exists and the stable tag does not.
2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from
`packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to
`:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`.
3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable
version (the `@sha256` wins regardless), restamp the version-stamped
assets (operator manifests, cozypkg, openapi; the heavy Talos assets are
copied verbatim from the rc draft), open a `release-X.Y.Z` PR.
4. Merging the PR reuses the existing release-PR e2e and
`pull-requests-release.yaml` finalize to cut the **write-once** stable
tag at the merge commit and publish the release. Squash is disallowed
(the tag needs a real merge commit).
`nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by
`NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line
(dry-run by default).
## Validation
**Local end-to-end — the core claim is proven, not asserted.** The whole
design rests on "copy-by-digest to a new tag preserves the digest, so
stable == the e2e-tested rc, bit-for-bit." This was exercised against a
real registry (`ttl.sh`) with real `skopeo`:
- Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp
tree exercising all three `values.yaml` digest shapes (single `image:`
string, split `repository`/`tag`/`digest` map, and the
`platformSourceRef` OCI artifact), then ran the actual
`hack/promote-retag.sh v1.4.0`.
- **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the
identical digest** (`sha256:fd8d9aa6…`), across all 17 platform
manifests. The script's own `skopeo inspect` post-check passed, and an
independent re-inspection confirmed it.
Local validation caught (and this PR fixes) two bugs that only surface
against a real registry — never in static lint:
1. `skopeo copy --multi-arch all --all` — mutually exclusive flags,
fatal error; every retag would have failed on CI. Now `--multi-arch
all`.
2. The same digest was retagged twice when an image appeared in two
value shapes; the ref set is now deduped on the canonical `repo@digest`.
**Other local checks:** `go test ./pkg/version` + `go build`/`vet`;
`helm template` renders `COZYSTACK_VERSION` in all 3 operator variants
(omitted when unset); `make manifests` stamps the version into the
install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint`
clean and `act -l` resolves the job graph on all workflows; the rc-tag
parser and nightly version-math unit-tested (accept/reject +
minor-vs-patch bumps).
**Still CI-only (cannot be exercised offline):** the full workflow
runtime — github-script API calls, OCIR auth, the rc-release /
staging-branch preconditions, the nightly `base_ref` push mechanics, and
self-hosted runners. The *logic* inside the steps is unit-tested; the
*orchestration* is not. Treat it as unproven until the workflows run.
## Out of scope / required follow-up (needs repo admin — not doable from
a PR)
- Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by
the CI app, no delete/update; "limit branches/tags updated in a single
push" = 1.
- Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo
variables.
- Registry-side pruning of `*-nightly.*` image tags (no OCIR delete
parity yet — tracked TODO in `retention.yaml`).
## Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added nightly publishing (mirror-by-digest, disk build, e2e
validation) and nightly retention pruning.
* Added RC-to-stable promotion via digest retagging (no rebuild).
* Operator/installer and console now expose version/platform metadata,
including runtime override via `COZYSTACK_VERSION`.
* **Bug Fixes**
* Enforced write-once tag behavior and fast-forward-only maintenance
updates across release workflows.
* Made nightly mirroring/selection and retention pruning more selective
and safer.
* **Tests**
* Added Bats coverage for nightly mirroring and RC retagging, plus new
Helm/unit checks.
* **Documentation**
* Updated the release model to center RC promotion and tag immutability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for the newer etcd snapshot workflow
(BackupClass/BackupJob/RestoreJob) with in-place restore.
* Updated etcd chart/operator packaging and migration flow for the
v1alpha2 operator release.
* **Bug Fixes**
* Switched readiness/condition handling to **Available**.
* Improved waits to cover all etcd member pods during restore and
scenario steps.
* **Documentation**
* Refreshed etcd backup/restore docs and example scenarios to match the
v1alpha2 APIs and parameters.
* **Tests**
* Updated e2e coverage to validate v1alpha2 operator runtime contracts
and replaced legacy backup-block tests with a snapshot round-trip test.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Promotion mutated the registry at dispatch time — retagging ~37 images to
vX.Y.Z (+:latest), publishing the stable cozy-installer chart — before the
promote PR's e2e ran and before any merge. An abandoned promotion then left
stable-named bytes for a version that never shipped, and the (correct) image
write-once check wedged any later re-promotion to the same version.
Move every irreversible registry side effect into pull-requests-release.yaml
finalize, which runs only after the promote PR merged (= after its e2e passed),
making promotion atomic. promote-rc.yaml now only stages the release-X.Y.Z
branch, drafts the stable release, and opens the PR; an abandoned promotion
leaves nothing but an overwritable draft + branch.
Folded into the same restructure:
- Cut the api/apps/v1alpha1/vX.Y.Z Go-module tag in finalize, at the stable
tag's commit. It was gated in tags.yaml on `release_exists==false &&
is_stable`, unsatisfiable under promotion (the draft always pre-exists), so
promoted stables never got their submodule tag — a regression for Go
consumers. Finalize is where the stable tag is born, so it cuts the matching
submodule tag write-once (stable only).
- Gate :latest by max-semver. promote-retag.sh now moves :latest only when
MOVE_LATEST=1 (default off); finalize passes it the same make_latest decision
the release gets, so a patch on an older line moves neither the release's
`latest` nor the images' :latest. Added MOVE_LATEST bats coverage.
- Stamp platformVersion into the published stable chart's default values
(uncommitted, at package time) so `helm --install --version X.Y.Z` reports the
stable version in telemetry instead of the rc version baked into the operator
image. Not committed to the branch — that would leak platformVersion=vX.Y.Z
onto main and mis-stamp the next rc.
- promote-rc: serialize all promotions (static concurrency group, not per-rc);
label the PR full-e2e so the full suite runs; scope the rc->stable sed to the
depth-2 values.yaml glob plus the kubernetes .tag files (the build's own write
set); tolerate a leftover draft so re-promotion is not wedged.
- finalize: assert the PR author is cozystack-ci[bot], and run on an ephemeral
oracle-vm runner (needs skopeo/helm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
collect_refs scans "$TREE"/*/*/values.yaml (depth-2), but the host rewrite
used `find "$TREE" -name values.yaml` (any depth). A values.yaml nested
deeper than group/pkg would then have its image host rewritten to GHCR
without its image ever being mirrored there, leaving a dangling ref. Rewrite
over the same depth-2 glob so the rewritten set is exactly the mirrored set
(this also matches the existing --dry-run message).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The backup round-trip test could wedge for 20 min and then report a bare
EtcdSnapshot phase=Started with no root cause: dump_diagnostics dumped the
(healthy) EtcdCluster and the operator log, but the snapshot agent runs in
its own Job Pod, so the etcd-read / S3-upload error lives only in that Pod's
log and in the BackupJob/EtcdSnapshot status.
- dump_diagnostics now dumps the whole backup chain: BackupJob/Backup and
EtcdSnapshot describe, each snapshot Job's describe + Pod logs (selected by
the operator's etcd-operator.cozystack.io/cluster label), and recent
namespace events (the primary signal when a hung Pod logs nothing).
- wait_for_field gains an optional terminal-failure value; steps 04/05 pass
Failed so the BackupJob/RestoreJob waits return immediately on a terminal
phase instead of polling to the timeout — which also keeps the snapshot
Job Pod within its 600s TTL so its log is still readable at dump time.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The v1alpha2 operator sets Available=True early (reason ClusterDiscovered)
as soon as the seed member forms the cluster, then scales the remaining
members up one at a time. Gating the runtime-contracts test on Available
raced the member-Pod count assertion and saw only 1/3 Pods, failing the
run. Gate on status.readyMembers=3 instead — the count of members whose
EtcdMember Ready condition is True (and the /scale status path), which
only reaches 3 once every member Pod is Running and serving. Bump the
wait timeout to 300s to cover sequential one-at-a-time rollout.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Address review feedback from coderabbitai on hack/e2e-apps/etcd.bats: read the container name, metrics port, and Pod IP with the strict member selector (name+instance+managed-by) plus a Running-phase filter instead of the broad app.kubernetes.io/name=etcd, so .items[0] cannot resolve to a non-member or not-yet-running Pod. Also assert a non-empty Pod IP before the metrics probe.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
kubernetes-previous.bats intermittently fails on
`kubectl wait deploy --for=condition=available` with
`Error from server: etcdserver: leader changed` even when the target
Deployment is milliseconds away from becoming Ready. Observed in run
28772259934 on head `5f0bec622`: three of the four deployments
already reported `condition met`, the fourth (the Kamaji tenant CP
Deployment) was ~200ms from Ready per the crust-gather snapshot
captured 20s later (`readyReplicas: 2`), and the wait exited
non-zero on the FIRST etcd-side error it saw on the watch channel.
Cause: the e2e sandbox management cluster is a 3-node kind on Talos
VMs with a 3-instance etcd HA cluster. By the time
kubernetes-previous starts (after kubernetes-latest,
kubernetes-oidc-customconfig, kubernetes-oidc-system, plus the
storageclass-fallback probe), accumulated CDI+DRBD IO pressure and
lingering "still attached to node" PVCs from prior tests push etcd
fsync latency high enough to trip heartbeat timeouts on the leader.
The election that follows resolves in seconds, but `kubectl wait`
sees the server-side error mid-watch and exits — clients would
normally reconnect, but `wait` treats a watch-channel error as
terminal.
Wrap the four management-cluster `kubectl wait` calls in
run_kubernetes_test with a retry helper that:
- Retries only on a curated allowlist of server-side transients
(`etcdserver: leader changed`, `etcdserver: request timed out`,
`the server was unable to return a response in the time
allotted`).
- Never swallows legitimate errors (NotFound, --timeout expired,
permission denied, etc.).
- Caps at 3 attempts with a 5s backoff so a persistently-broken
etcd still surfaces loudly.
Untouched:
- `kubectl wait namespace tenant-test --timeout=20s` at line 308 —
short-lived, hits mgmt but the object always exists ~instantly.
- `kubectl wait deployment ... --kubeconfig tenantkubeconfig-...` at
line 509 — waits on the TENANT apiserver's own etcd, a different
failure domain (single-instance kine or CNPG-backed via Kamaji,
not the 3-instance mgmt etcd that flapped).
- Other bats files with mgmt-cluster `kubectl wait` — out of scope
for this PR; if the same flake reappears there a follow-up can
lift the helper into a shared library.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The audience-list assertion used \`grep -qF "- tenant-test-..."\`.
Because the pattern begins with a dash, grep treats the leading \`-\`
as an option flag, aborting the test with \`grep: invalid option --
' '\` before the actual match ever runs. Insert an explicit \`--\`
end-of-options separator so the YAML list-item pattern is passed as a
pattern rather than a flag.
Custom-config counterpart is unaffected — its grep patterns do not
start with a dash.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
cozytest.sh (the e2e runner) is not real bats — it never invokes
setup()/teardown(). The OIDC files were structured around those hooks
to seed TEST_NAME and cleanup the tenant CR between tests, so with
cozytest strict-mode wrapping (\`set -eu\`), the first \${TEST_NAME}
expansion inside an @test aborted with 'parameter not set' before the
apply could even run.
Rework both files to the convention documented in external-dns.bats:
seed TEST_NAME at file scope (no per-test uniqueness needed — the
tenant-test namespace is single-suite), keep cleanup_kr() as a shared
helper, add a file-level cozy_cleanup() wrapper (the hook cozytest
actually calls on suite exit / first-failure), and move the pre-test
CR delete inline at the top of the first @test that creates the CR.
Subsequent @tests only read state and rely on the CR persisting until
cozy_cleanup.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Three items from the gemini-code-assist and coderabbitai passes that
survived the Phase-1 redesign:
- `oidc-rbac-job.yaml` (gemini HIGH): pre-delete cleanup uses
`kubectl --request-timeout=30s` in addition to `--timeout=30s`
on the delete verb, so an unreachable tenant apiserver cannot
stall the pre-delete hook on the HTTP round-trip itself. The
`|| echo WARNING` fallback still keeps a truly unreachable
cluster from blocking `helm uninstall`.
- `oidc-keycloak.yaml` + `_helpers.tpl` (gemini LOW): switch
`_cluster.{oidc-enabled,root-host}` reads to `dig` with an empty
map default so a missing `_cluster` key nil-safely emits an
empty string and hits the platform-OIDC guard instead of a
template panic on `.Values._cluster`.
- `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`
(coderabbit MAJOR): add an existence-poll `timeout ... until
kubectl get secret ... ; do sleep ...; done` before the
`kubectl wait secret --for=jsonpath` line. Without it a runner
that reaches the wait before the HR has rendered the Secret
gets an immediate `NotFound` and the test flakes.
Adds one helm-unittest case covering the nil-safe `_cluster` path.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the design proposal explicitly lists TWO bats files:
`kubernetes-oidc-system.bats` and `kubernetes-oidc-customconfig.bats`.
The first revision shipped only one combined file; this commit splits
them.
The CustomConfig bats covers what System cannot: the
operator-supplied AuthenticationConfiguration is plumbed onto the
KamajiControlPlane, the resulting Secret carries the BYO issuer (and
emphatically NOT a `cozy` issuer), nothing lands in the `cozy` realm,
and no OIDC kubeconfig Secret is pre-rendered (the operator
distributes their own).
The System bats header is updated to call out the deferred parts
(in-tenant CRBs / view-vs-admin RBAC behavior / browser-flow login)
that need a full Kamaji bringup, which is already covered for the
cluster path by `kubernetes-latest.bats`.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fix: the design proposal specifies
`name: authentication-config` and
`mountPath: /etc/kubernetes/authentication-config`, but the previous
revision had renamed both to `cozy-oidc-authn-config` and
`/etc/kubernetes/cozy-oidc` to dodge a collision with the volume
name used in `apiserver_authentication_test.yaml`. That test was
demonstrating the GENERIC user-supplied `extra*` passthrough — not
the OIDC use case — so rename the test fixture instead and restore
the design-spec names on the chart side.
The reservation guard against user-supplied volumes reusing
`authentication-config` only fires when `spec.oidc.mode` is not
None, so clusters with OIDC off retain the previous freedom to name
their own volumes anything.
Renames in `apiserver_authentication_test.yaml`: the user-supplied
volume name `authentication-config` becomes `tenant-byo-mount`; the
Secret name `tenant-authentication-config` and the apiserver flag
`--authentication-config=` are untouched.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Self-review fixes for four linked lifecycle issues:
- The post-install Job created `<release>-oidc-kubeconfig` via
`kubectl apply` from inside the script; the resulting Secret had
no Helm release labels and was orphaned on `helm uninstall`. Now
a release-owned placeholder Secret is rendered up front in
`oidc-kubeconfig-secret.yaml` and the Job patches its `data`
field at runtime (the tenant CA still has to come from the
Kamaji-issued admin kubeconfig, which appears asynchronously).
- The ServiceAccount/Role/RoleBinding for the bootstrap Job were
hook resources with `helm.sh/hook-delete-policy: before-hook-creation`,
so they leaked on `helm uninstall`. They are now regular release
resources (no hook annotations) and are reaped by `helm uninstall`
alongside everything else. Both the post-install bootstrap Job
and the pre-delete cleanup Job find a working SA.
- The previous design tried to add `hook-succeeded` to the SA, but
that would delete the SA right after the bootstrap hook
succeeded — leaving the pre-delete cleanup Job with no SA to
schedule. Release-owned resources sidestep this race entirely.
- The Role had `verbs: ["create"]` on `secrets` without
`resourceNames` (Kubernetes RBAC cannot scope `create` by name).
With the placeholder Secret pre-rendered the Job only needs
`get`/`patch` on the specific name — the broad `create` rule is
removed.
Also tightens the script: `set -euo pipefail` (was `set -eu`, which
silently swallowed `kubectl get` failures inside command
substitutions) and uses `kubectl patch --type=merge` against the
existing Secret instead of `kubectl create --dry-run | kubectl apply`.
Adds three helm-unittest cases for the placeholder Secret across all
three modes.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Validate on a live cluster that:
- the new `spec.oidc.*` fields are admitted by the Kubernetes CRD;
- cozystack-api accepts the updated schema and renders a HelmRelease;
- the KamajiControlPlane gains `--authentication-config` and the
chart-owned `cozy-oidc-authn-config` mount and volume;
- the AuthenticationConfiguration Secret carries the cozy realm
issuer URL and the per-cluster audience;
- per-cluster `KeycloakClient` + audience `KeycloakClientScope`
land in the cozy realm (with the EDP operator present).
The full end-to-end RBAC flow (Job → CRBs inside the tenant cluster)
needs a Kamaji bringup, which kubernetes-{latest,previous}.bats
already cover for the cluster path; the OIDC browser flow is
explicitly out of scope per the design proposal. This file covers
only what the render-side regression budget can pay for.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Address review feedback from coderabbitai on hack/promote-retag.sh: the loop
copied each rc digest onto ${repo}:${STABLE} unconditionally, so a partial
prior run or a manual push could mutate already-released stable bytes before
the GitHub release/tag guards catch it. Inspect the destination first — no-op
when it already points at the rc digest (idempotent re-run), fail when it
points elsewhere. :latest stays mutable.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/nightly-mirror.sh copies the latest main build's cozystack-owned
component images from the CI build registry (OCIR) to the public release
registry (GHCR) by digest — no rebuild — then rewrites the baked package
tree's image hosts so the GHCR install closure is self-contained. It
reuses promote-retag.sh's three-shape ref enumeration, filters to
cozystack-owned refs, and deliberately excludes the cozystack-packages
artifact (rebuilt downstream from the rewritten tree).
Dry-run bats test under the cozytest harness guards the ref selection,
the cozystack-packages exclusion, third-party filtering, and the host
rewrite plan. shellcheck clean.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The CI workflow exports REGISTRY=<OCIR build registry> for every job
(.github/workflows/pull-requests.yaml). The dry-run test inherited it,
so promote-retag.sh filtered the tree's digest refs by the OCIR registry
— but the committed release tree vendors them under ghcr.io/cozystack/
cozystack (the script default), so the selector matched nothing and
aborted with 'No cozystack-owned digest-pinned image refs found',
failing the Unit & controller tests job.
Run the script with 'env -u REGISTRY' so it uses its default registry —
the one the tree's refs and this test's assertions actually use — and
add a slim self-diagnosing fallback that dumps the script's stderr and
the yq build on any unexpected non-zero exit.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/promote-retag_test.bats was written for upstream bats (run,
$status, $output, setup(), skip), but CI runs hack/*.bats through
hack/cozytest.sh, a pure-shell runner that has none of those. The first
'run' therefore died with 'cozytest.sh: run: not found', failing the
Unit & controller tests job.
Rewrite both cases in the cozytest idiom already used by
select-e2e_test.bats and check-host-runtime.bats: redirect to temp
files, capture the exit code with '|| rc=$?' so the harness's set -e
does not abort the expected-failure case, and assert with plain shell.
The selector logic under test is unchanged.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
collect_refs scraped every @sha256 ref from the package values.yaml, including
third-party images (docker.io/clastix/kubectl, ghcr.io/kvaps/...), bare upstream
tags (kube-ovn/keycloak/kilo) and a "--migrate-image=..." arg string. The first
skopeo copy to a registry CI cannot push to aborted the whole promotion under
set -e, so stable promotion could not complete. Filter the canonical refs to
those under $REGISTRY (default ghcr.io/cozystack/cozystack, matching
common-envs.mk; overridable for forks). Also require skopeo only when not
--dry-run, so the selection is testable without it.
Add hack/promote-retag_test.bats: a --dry-run over the real packages tree must
emit only cozystack-owned refs (zero third-party), and a REGISTRY override that
matches nothing must fail rather than promote the wrong set.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on hack/promote-retag.sh:27: command -v yq only confirms some yq is on PATH; the script's queries use mikefarah syntax, so reject python-yq and other variants by checking the version string, mirroring the Makefile build-deps check.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from gemini-code-assist on hack/promote-retag.sh:48: ref_repo now strips the :tag from the last path component only, so a registry host carrying a :port (e.g. localhost:5000/cozystack/operator) on a no-tag ref is preserved instead of truncated to the host.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add the rc->stable promotion path. A stable release is a renamed
release-candidate: the bytes shipped as vX.Y.Z are bit-for-bit the bytes
built and e2e-tested as vX.Y.Z-rc.N.
- hack/promote-retag.sh: reads the digest-pinned image refs the rc baked
into packages/*/*/values.yaml (single-string, split repository/tag/digest,
and the cozystack-packages OCI artifact) and `skopeo copy`s each — by
digest — to the stable tag and :latest. Copying by digest guarantees the
stable image is identical to the rc; the result is verified with
`skopeo inspect`. Validated offline with `sh -n` and the yq extraction
against the current tree; the skopeo path is CI-validated only.
- promote-rc.yaml: workflow_dispatch(rc_tag). Validates the rc exists and
the stable tag does not, retags via the script, rewrites the cosmetic
-rc.N substring in the vendored tags to the stable version (the @sha256
wins regardless), restamps the version-stamped assets (operator manifests,
cozypkg, openapi; the heavy Talos assets are copied from the rc draft),
and opens a release-X.Y.Z PR. Merging it reuses the existing release-PR
e2e + pull-requests-release.yaml finalize to cut the write-once stable tag.
Squash is disallowed on the PR (the stable tag needs a real merge commit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Extends `hack/cozyreport.sh` linstor module to copy
`/var/log/linstor-satellite/ErrorReport-*.log` off every
`linstor-satellite` pod into `linstor/error-reports/<node>.tgz` in the
cozyreport archive.
### Why
The linstor-Satellite pod stdout log references its error reports only
by ID:
```
[DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to create zfsvolume [Report number 6A4A394E-E4EA0-000000]
```
The report body (actual `zfs create -V …` stderr, DRBD adjust output,
storage-driver stack trace, exit code) lives on the satellite pod's
ephemeral filesystem at `/var/log/linstor-satellite/ErrorReport-*.log`.
crust-gather only captures pod stdout, so the report body is lost when
the e2e sandbox is torn down at end-of-job.
Consequence for post-mortem of `ResourceExhausted` / `Not enough
available nodes` CreateVolume retry loops (see e.g. PR #3044 CI
history): cozyreport shows only the driver-level symptom. Root cause
distinction (out-of-space vs dataset name conflict vs ZFS module error
vs kernel signal) is not possible from the artefact alone, and the
sandbox is unrecoverable by that point.
### Scope
`hack/cozyreport.sh` only, +32 lines under the existing `linstor module`
block. Guarded so:
- a satellite with zero reports (healthy case) leaves nothing in the
tree
- a transient exec / tar failure never fails the surrounding cozyreport
run
- empty parent directory is pruned
### Verified on a live cluster
- Label selector `app.kubernetes.io/component=linstor-satellite` matches
all 3 satellite pods on the dev cluster used for verification.
- Bundle for one active satellite comes out at ~60 KiB (dozens of
historical reports tarred).
No product-side change, no dependency on capture happening during a
specific test phase; runs as part of the existing end-of-job cozyreport
collection.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced offline LINSTOR diagnostics collection by adding an
error-report index and bundling controller and satellite error-report
logs into downloadable archives.
* Generates per-node satellite bundles using stable node naming for
clearer organization of collected diagnostics.
* **Bug Fixes**
* Avoids creating empty or invalid error-report archives and
automatically cleans up unused bundles to prevent clutter in the
collected output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
kubernetes-previous.bats intermittently fails with `zfs create -V ...
returned with exitcode 1: cannot create '...': out of space` on a
satellite even though the LINSTOR pool measured 30+ minutes later shows
100+ GiB free per satellite. Root cause proven from a satellite
ErrorReport captured in the cozyreport artefact (PR #3044 run
28751310913, report id 6A4AADFD-349B2-000000): the previous tenant
Kubernetes test's cozy_wait_tenant_drained gate returned as soon as
its PVCs disappeared from the API, but the satellite-side ZFS `zvol
destroy` for the just-deleted replicated (autoPlace=3, DRBD) 20 GiB
worker disks lagged tens of seconds behind that signal as DRBD
adjusted, unref counts drained, and ZFS batch-destroyed the datasets.
The next tenant test's CreateVolume then landed inside that window
against a pool that had not yet reclaimed the space, LINSTOR-CSI
retried autoplace against its own still-being-torn-down previous
placement, and worker Machine bringup stretched past the MHC
nodeStartupTimeout budget with no cluster-side fault.
Add a companion `cozy_wait_linstor_pool_free` helper that polls the
LINSTOR controller for the smallest FreeCapacity across every ZFS
satellite pool and blocks until it reaches a per-satellite threshold
(default 90 GiB — covers the ~82 GiB worst-case footprint of two
20 GiB replicated worker targets across all three satellites plus two
21 GiB CDI scratch PVCs landing on the same node via the local
storageClass). Called from cozy_cleanup immediately after
cozy_wait_tenant_drained. Bounded and best-effort: 300 s timeout,
`|| true` at the call site, warning printed with the observed
pool state if the threshold is not reached in budget so the next test
does not misdiagnose the pressure.
The threshold is a floor for the concrete failure mode observed in the
E2E suite; the surrounding kubernetes-latest -> kubernetes-previous
handoff is the only run in the current suite that immediately follows
a heavy tenant teardown with another tenant provisioning, so this
gate is the natural place for the check. E2E-scope only, no product
or chart change.
The proximate observation (`zfs create out of space`) surfaced because
of PR #3209 which taught hack/cozyreport.sh to bundle satellite
ErrorReports; without that artefact the root cause was ambiguous
between multiple plausible LINSTOR internal races. Ships alongside
that PR.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The linstor-Satellite pod stdout log references its error reports
only by ID:
[DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to
create zfsvolume [Report number 6A4A394E-E4EA0-000000]
The body of that report (which carries the actual `zfs create -V …`
stderr, the DRBD adjust output, the storage-driver stack trace, and
the exit code) lives at `/var/log/linstor-satellite/ErrorReport-*.log`
on the satellite pod. crust-gather only captures pod stdout, so the
report body stays on the pod's ephemeral filesystem and is lost when
the e2e sandbox is torn down at end-of-job.
Consequence for post-mortem: a run that failed on a
`ResourceExhausted` / `Not enough available nodes` CreateVolume retry
loop shows only the driver-level symptom in cozyreport. Root cause
attribution (out-of-space, dataset name conflict, ZFS module error,
kernel signal) is not distinguishable without the report body, and
without the sandbox is unrecoverable.
Extend the linstor module in `hack/cozyreport.sh` to tar the
`ErrorReport-*.log` set off every `linstor-satellite` pod into
`linstor/error-reports/<node>.tgz`. Guarded so a satellite with zero
reports (the healthy case) leaves nothing in the tree, and a
transient exec / tar failure never fails the surrounding cozyreport
run.
Verified against a live dev cluster: label selector
`app.kubernetes.io/component=linstor-satellite` matches all three
satellite pods, and the tar bundle for one active satellite comes
out at ~60 KiB. No product-side change.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The E2E sandbox tenant (tenant-test) provisions back-to-back tenant
Kubernetes control planes via kubernetes-latest.bats and
kubernetes-previous.bats. Each control plane brings up two worker VMs
whose 20Gi root disks are populated by CDI, and CDI's second-phase
importer allocates a scratch PVC of the same size before the qcow2
unpack. Peak per-tenant footprint is therefore ~82Gi during import
(2 × 20Gi target + 2 × 21Gi scratch), collapsing to 40Gi at rest.
kubernetes-latest teardown deletes the tenant Kubernetes CR and waits
via cozy_wait_tenant_drained for the KubeVirt VMs, VMIs and namespace
PVCs to disappear. The API-level PVC delete returns quickly, but the
LINSTOR CSI detach + DRBD resource release on the underlying nodes
lags past the 300s drain budget. During that window, the deleted
PVCs still count against ResourceQuota's requests.storage because
the storage-provisioner has not finished releasing the volume — so
kubernetes-previous starts its own CDI import with 40Gi of stale
kubernetes-latest quota still charged, and CDI's scratch PVC create
for the second worker (used=80Gi + scratch=21Gi = 101Gi) is
rejected by the tenant-quota admission check at 100Gi. The second
worker's DataVolume never leaves ImportInProgress, the tenant node
never joins, and the 12-minute node-join deadline in
run-kubernetes.sh trips kubernetes-previous.bats.
Raise the sandbox tenant-quota storage ceiling to 200Gi so the
overlap between one test's slow storage cleanup and the next test's
CDI import fits inside the quota. The failing-run diagnostic is
cozyreport artifact from run 28730757213:
cdi-deployment log:
scratch PVC API create errored: … forbidden: exceeded quota:
tenant-quota, requested: requests.storage=22763536384,
used: requests.storage=85899345920,
limited: requests.storage=100Gi
This is a test-infrastructure fix; the 100Gi default in the tenant
chart itself is unchanged. The assertion below the CR update is
lifted from 100Gi to 200Gi in lockstep.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The Gateway (cozystack-gateway-hostname-policy) and Route
(cozystack-route-hostname-policy) admission policies constrain tenant
hostnames on the Gateway API path. Gateway API is opt-in and off by
default, so in the default configuration tenant applications publish
through a legacy networking.k8s.io/Ingress on the shared ingress-nginx,
which had no equivalent policy.
Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in
tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[]
is allowed when it is within the namespace's own apex
(namespace.cozystack.io/host), or when it is a concrete (non-wildcard)
name entirely outside the platform root apex (_cluster.root-host); a
hostname under the platform root apex but outside the namespace's own
apex, and a wildcard taken through the outside-root path (which could
match the platform apex itself), are rejected. The outside-root branch
keeps the kubernetes app's Proxied exposeMethod working, since it routes
a user-supplied external domain to a nested cluster. A rule with no host
(absent or empty, an unbounded catch-all) and spec.defaultBackend are
also rejected.
The policy templates the platform root apex into its CEL and renders
only when _cluster.root-host is set, so it is never shipped in a state
that would admit every hostname. A tenant-* namespace missing its own
apex label is denied.
Cover the policy with helm-unittest (render, including the root-host,
empty-host, and wildcard guards) and behavioral admission tests against
the live apiserver (in-apex allow, external-domain allow,
own-apex-wildcard allow, external-tls allow, under-root-outside-apex
deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny,
hostless deny, empty-string-host deny, defaultBackend deny), and document
it as an additional layer in the gateway security model.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The failure-path dump queried mariadbs.k8s.mariadb.com with $name, but the
operator CR is named after the Helm release ({{ .Release.Name }} = mariadb-$name),
so the dump would hit NotFound instead of showing the rejected CR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.
Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:
- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
external access via `service.type: LoadBalancer` and keep `primaryService`
LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
(no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
-primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
installs are visible in the dashboard.
Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
## What this PR does
The `Configure Tenant and wait for applications` install test brings up
the entire root-tenant monitoring stack and waits for each component to
become ready/operational. Those waits span the VictoriaMetrics operator
resources (`vmalert`/`vmalertmanager`, `vlclusters/generic`,
`vmcluster/shortterm`+`longterm`) and the Grafana pair (`grafana-db`
CNPG cluster, `grafana-deployment`). Each converges only once its
underlying workloads schedule and turn Ready, and during platform
bring-up they contend for the same node resources as the rest of the
install — so their convergence is load-sensitive.
Only `vmalert`/`vmalertmanager` used a 15m budget; the other four used
10m, so this block carried a 10m/15m split even though all its resources
contend for the same node capacity during install. Under install-time
load (multiple concurrent e2e sandboxes on one runner) the monitoring
bring-up is slow across the whole block rather than on a single
resource, so the shorter budgets risk failing the whole install on PRs
whose branches never touched monitoring — and per-resource bumps are
whack-a-mole: raising one budget just moves the risk to the next
component in the block. On a calm sandbox each of these reaches
ready/operational in under a second, which is what distinguishes the two
cases — slow-not-stuck convergence under contention, not a product
regression.
This lifts the whole tenant-root monitoring wait-set to a single uniform
15m budget and wraps each wait in an on-timeout diagnostic block that
dumps the resource YAML (status conditions / `updateStatus`) plus
tenant-root pod state. A genuinely stuck resource still fails — now at
15m, and with legible diagnostics instead of a bare `timed out` line —
so this does not weaken the gate's ability to catch real breakage. The
extra budget costs near-zero wall-time on the happy path and stays well
inside the E2E job's 180m cap.
The remaining upstream angle is the real driver: how long the
VictoriaMetrics/VictoriaLogs stack and the Grafana bring-up take to
become operational under resource contention during install. This change
addresses the e2e robustness side; reducing that bring-up cost is a
separate, product-side improvement.
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved and standardized readiness checks during tenant setup so
application bring-up waits behave more consistently.
* Enhanced failure handling to provide clearer diagnostics (including
resource details and pod status) when readiness checks time out.
* **Tests**
* Standardized longer readiness wait timeouts for multiple application
components to reduce flakiness during e2e runs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Fixes the dominant flake in the tenant-Kubernetes e2e suite, where a run
fails on worker-node-join under sandbox memory/storage pressure.
The two heavy Kamaji tenant tests — `kubernetes-latest` then
`kubernetes-previous` — run back-to-back in one CI sandbox, each
provisioning a tenant cluster with KubeVirt worker VMs. The inter-test
cleanup hook (`cozy_cleanup` in `hack/e2e-apps/run-kubernetes.sh`)
deletes the tenant `Kubernetes` CR and waits only for the CR to
disappear. That wait finalizes as soon as the CR's finalizers clear,
which merely *triggers* teardown of the CAPK worker VMs and their
DataVolume-backed disk PVCs — it does not await it. The virt-launcher
pods keep their guest RAM reserved until the VMIs are gone, so the next
tenant test's worker VMs begin scheduling against a sandbox the previous
tenant has not yet vacated. Under the constrained sandbox this starves
memory, a worker VM misses the node-join budget, and the
MachineHealthCheck cannot remediate when both workers are unhealthy — so
the run flakes on worker-node-join.
This adds a bounded drain barrier to `cozy_cleanup`: after deleting the
CR it now blocks until the tenant's VirtualMachines,
VirtualMachineInstances and disk PVCs have actually drained from the
`tenant-test` namespace, so the next test starts on a freed sandbox.
The wait is bounded (5m) and best-effort: cozytest already runs
`cozy_cleanup` wrapped in `|| true`, and the waiter returns loudly
(warning plus a leftover dump) on timeout, so a stuck teardown can never
hang the job past the deadline — it leaves the sandbox no worse than
before. The probe is namespace-scoped: `tenant-test` is provisioned with
etcd/monitoring/seaweedfs disabled so it carries no baseline PVCs, and
the e2e apps run sequentially each cleaning up after themselves, so the
only VMs/VMIs/PVCs present belong to the tenant being torn down.
The pure drain exit-condition (`cozy_tenant_drained`) is factored out
and unit-tested in `hack/run-kubernetes-drain_test.bats` (runs under
`make unit-tests`), including the invariant that a failed probe's `err`
sentinel is never misread as a completed drain.
### Screenshots
Not applicable — e2e harness change, no UI.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved Kubernetes test cleanup so tenant resources are fully drained
before the next test starts, reducing leftover VMs, VM instances, and
PVCs.
* Added more reliable timeout handling and clearer diagnostics when
teardown or node readiness checks fail, making flaky test issues easier
to identify.
* **Tests**
* Added coverage for tenant-drain detection to verify empty,
whitespace-only, multi-resource, and error scenarios behave as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Configure-Tenant install test completes the tenant-root monitoring
bring-up by waiting for the grafana-db CNPG cluster and the
grafana-deployment Deployment. Both used a 10m budget while the
VictoriaMetrics waits above them use 15m. They complete the same
monitoring bring-up and contend for the same node resources during
install, so under install-time load (concurrent e2e sandboxes on one
runner) either can be slow, and the 10m/15m split within the same
contend-for-resources bring-up risks failing the whole install on a PR
that never touched monitoring.
Raise both Grafana waits to the same uniform 15m budget as the
vm-operator waits and wrap them in an on-timeout diagnostic block that
dumps the resource YAML plus tenant-root pod state, so a genuinely stuck
resource still fails at 15m with legible diagnostics instead of a bare
"timed out" line. The extra budget costs near-zero wall-time on the happy
path and stays inside the E2E job budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
vlclusters/generic and vmcluster wait for updateStatus=operational with
a 10m budget, while their vmalert/vmalertmanager sibling in the same
block already uses 15m. All three are vm-operator-managed monitoring
resources that only become operational once their workloads schedule and
turn Ready, and during platform bring-up they contend for node resources
with the rest of the install. On a calm sandbox each converges in under a
second, but under install-time load (concurrent e2e sandboxes on one
runner) monitoring bring-up is slow, and a 10m/15m split inside a block
whose resources all contend for the same node capacity risks failing the
whole install on a PR that never touched monitoring.
Unify vlclusters and vmcluster on the sibling's 15m budget so the whole
block shares one headroom, and dump resource YAML plus pod state on
timeout across all three waits so a genuine stuck-not-slow regression
stays legible instead of a bare "timed out" line. The extra budget costs
near-zero wall-time on the happy path and stays inside the E2E job
budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What
The e2e install pipeline still flakes on a loaded runner, failing at a
different point each run. Root-caused via repeated runs and a live SSH
breakpoint; the platform itself is fine (the components do converge),
the failures are slowness + self-inflicted churn under resource
pressure.
### Fixes
1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory
limit was `110Mi`. The updater watches every pod in the cluster, so on a
full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the
HelmRelease "early due to stalled resources", cascading NotReady to
monitoring, velero, etcd-operator and everything downstream. Raised to
`512Mi` (request `200Mi`).
2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA
`VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA
recommender(s) during install and adding pod churn on an already busy
cluster. Switched to `Initial`, matching the existing monitoring/etcd
`VPA->Initial` fixes.
3. **Install wait budgets** — components reach readiness but overshoot
tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant
aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`.
(The crust-gather failure-snapshot timeout that kept whole jobs hanging
to the 3h cancel is already on `main`.)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased several installation and readiness wait times to better
handle slower environment startup and reduce premature failures.
* Updated vertical autoscaling behavior to avoid unnecessary pod
restarts during deployment.
* Raised resource limits for the autoscaling updater to improve
reliability under load.
* **Tests**
* Aligned automated checks with the new autoscaling behavior and longer
readiness expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The Cilium leak healer (`hack/e2e-cilium-endpoint-leak-healer.sh`, the
interim CI band-aid for the Cilium endpoint-leak class) escalates an
unhealed leaked-IP reuse to a Tier-2 `cilium-agent` restart. For one
variant that restart is both futile and harmful.
When the disputed IP is the node's per-node Cilium ingress IP
(`CiliumNode.spec.ingress.ipv4`), it is held by the `reserved:ingress`
endpoint — allocated from the node pod CIDR whenever Gateway API / Envoy
is enabled (`enable-envoy-config`,
`daemon/infraendpoints/infra_ip_allocation.go`). Restarting the agent
cannot clear it: the agent re-derives the same ingress IP from the
CiliumNode spec on every boot. Worse, the restart can re-trigger the
race that caused the wedge in the first place — it rebuilds the
host-scope (`kubernetes` IPAM) allocation bitmap empty
(`pkg/ipam/hostscope.go`; `RestoreFinished()` is a no-op), then
re-reserves the ingress IP via a path decoupled from pod-endpoint
restore, so a CNI ADD served in that window can be handed the ingress IP
again. That is the infra-IP-vs-pod-IP double-allocation behind the "IP
already in use" sandbox rejection.
This is an upstream Cilium bug in host-scope IPAM with Gateway API; no
released Cilium fixes it as of v1.19.5 (the deployed version). The
complete fix belongs upstream — the agent must reserve
`spec.ingress.ipv4` in the host-scope allocation bitmap before the CNI
server serves allocations, or reconcile it with the restored
`reserved:ingress` endpoint, or source it from a dedicated range. That
is tracked upstream in cilium/cilium#46799.
This PR makes the CI healer correct for that variant: it detects an
exact match of the wedged IP against the node's
`CiliumNode.spec.ingress.ipv4` and skips the Tier-2 restart — surfacing
the case and rescheduling the wedged pod best-effort instead. The match
is gated so an empty ingress IP (CiliumNode absent or RBAC denied) never
matches, so a false positive can never suppress a restart that would
clear a real in-memory leak. The change adds a `ciliumnodes` read to the
healer RBAC and a unit test pinning the exact-match-and-non-empty
contract.
### Screenshots
N/A — CI test-infrastructure change, no UI impact.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced endpoint leak healer to skip Tier-2 agent restarts when the
disputed IP exactly matches a node’s reserved per-node Cilium ingress
IP.
* Added lightweight unit tests for the new IP-matching behavior.
* **Bug Fixes**
* Prevents unnecessary remediation when the disputed IP is empty or
doesn’t exactly match the node ingress IP.
* When a match is detected, the healer deletes the affected wedged pod
as a best-effort reschedule step.
* **Chores**
* Updated healer permissions to read Cilium node ingress information
cluster-wide.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
When a tenant Kubernetes cluster hits the 12m node-Ready deadline with
fewer than two nodes joined, the failure path dumped only `describe
nodes` and `get hr`. That cannot tell apart the two node-join failure
sub-modes: a worker VM that never booted (virt-launcher Pending or
OOMKilled) versus a VM that booted fine but whose kubelet never
registered a Node. The tenant cilium-operator HelmRelease reports
"InProgress" identically in both, so the current dump is not enough to
diagnose the failure or design a fix.
Dump scoped, guarded diagnostics before failing fast:
- worker VM/VMI/virt-launcher state on the management cluster, which is
what actually splits the two sub-modes;
- tenant kubelet CSRs plus the talos-csr-signer sidecar log, which reveal
a boot-but-no-Node stall.
In-guest Talos/kubelet capture is deliberately left out: talosctl needs a
tenant-cluster talosconfig the runner does not have (the tenant workers
use a separate Talos CA from the management sandbox), so the gap is
documented in the failure path rather than shipped as a call that would
fail mTLS and capture nothing.
Every capture is guarded with `|| true` so a capture failure never masks
the real exit.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The e2e install still flakes on a busy runner where components reach
readiness but overshoot tight wait budgets:
- keycloak HR chain (keycloak-db CNPG -> keycloak StatefulSet ->
keycloak-configure) reaches Ready ~15-18m in: 10m -> 20m.
- tenant aggregate wait (hr/etcd hr/ingress hr/monitoring hr/seaweedfs
hr/tenant-root): the tenant stack only starts creating pods ~9-10m in
and then needs several more minutes: 10m -> 20m.
- per-app tenant readiness waits (ingress, etcd, VictoriaLogs/Metrics
clusters, grafana-db, grafana): 5m -> 10m.
Diagnosed via a live SSH breakpoint: the HRs all do reach Ready, just
outside the old budgets.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The inter-test cleanup hook deletes the tenant Kubernetes CR and waits
only for the CR to disappear. That wait finalizes as soon as the CR's
finalizers clear, which merely TRIGGERS teardown of the CAPK worker VMs
and their DataVolume-backed disk PVCs -- it does not await it. The
virt-launcher pods keep their guest RAM reserved until the VMIs are
gone, so the next back-to-back tenant test's worker VMs begin scheduling
against a sandbox the previous tenant has not yet vacated. Under the
constrained CI sandbox this starves memory, a worker VM misses the
node-join budget, and the run flakes on worker-node-join (the
MachineHealthCheck cannot remediate when both workers are unhealthy).
Block cozy_cleanup until the tenant VMs, VMIs and disk PVCs have
actually drained from tenant-test before it returns, so the next test
starts on a freed sandbox. The wait is bounded (5m) and best-effort:
cozytest runs cozy_cleanup wrapped in `|| true` and the waiter returns
loudly on timeout, so a stuck teardown can never hang the job longer
than the deadline. The probe is namespace-scoped -- tenant-test is
provisioned with etcd/monitoring/seaweedfs disabled so it carries no
baseline PVCs, and e2e apps run sequentially each cleaning up after
themselves, so the only VMs/VMIs/PVCs present belong to the tenant being
torn down.
The pure drain exit-condition (cozy_tenant_drained) is unit-tested in
hack/run-kubernetes-drain_test.bats, including the invariant that a
failed probe's "err" sentinel is never misread as a completed drain.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Every `*.bats` file in the repo runs through `hack/cozytest.sh`, not
real bats — the e2e files via `packages/core/testing/Makefile` and the
`hack/*.bats` unit tests via `make bats-unit-tests`. cozytest runs each
`@test` under `set -e`, and POSIX errexit is suppressed for a pipeline
beginning with `!`. So a bare `! cmd` must-not-happen assertion never
fails the test: when the forbidden condition actually occurs (the
command succeeds), the negated pipeline returns non-zero but errexit
ignores it and the test passes anyway. Ten such assertions across six
files were silently vacuous.
This converts each to `if cmd; then echo "<reason>"; false; fi`. The
`if` condition is exempt from errexit (so the probe itself never aborts
the test), and the `false` in the taken branch is a plain command under
`set -e`, so it aborts and fails the test with a legible message when
the forbidden condition holds. This is the must-not-happen inverse of
the `cmd || { echo …; false; }` pattern already used in the suite (e.g.
harbor.bats, per docs/agents/e2e-testing.md). Original redirections and
quoting are preserved, so happy-path behavior is unchanged — only the
failure path now actually fails.
Converted assertions:
- `bucket`: a read-only user must not be able to upload (access-control
regression guard)
- `gateway`: a child tenant must not have its own gateway HelmRelease
(inheritance)
- `kuberture`: external-dns public/internal hostname isolation
- `e2e-install-cozystack`: tenant-name validation must reject rather
than warn; the version ConfigMap must be gone after delete
- `build-matrix` (unit test): talos/installer must be excluded from the
parallel matrix
- `capture-dataplane` (unit test): kube-dns and pending rows must be
filtered out
A separate in-flight etcd test change owns two more such assertions in
`etcd.bats`; they are left out of this PR to avoid a conflicting edit.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Strengthened several end-to-end/Bats assertions to fail more
explicitly when expected cluster conditions aren’t met.
* Improved matrix validation to reliably confirm excluded components are
absent.
* Updated dataplane load balancer filtering, bucket readonly upload, and
gateway inheritance checks to avoid false/ambiguous pass cases.
* Tightened DNS hostname isolation probes and cozystack installer
validations (tenant name dashes, deletion protection) with clearer
“expected vs found” failures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
These bats files are all run by hack/cozytest.sh, which executes each
@test under `set -e`. POSIX errexit is suppressed for a pipeline that
begins with `!`, so a bare `! cmd` must-not-happen assertion never fails
the test: when the forbidden condition actually occurs (cmd succeeds) the
negated pipeline returns non-zero but errexit ignores it and the test
passes anyway. These assertions were silently vacuous.
Replace each with `if cmd; then echo "<reason>"; false; fi`, the idiom
already used and documented elsewhere in these files — the if-condition
is exempt from errexit and the `false` in the taken branch fails the test
loudly. The forbidden condition now fails the test as intended:
- bucket: read-only user must not be able to upload (access control)
- gateway: child tenant must not have its own gateway HelmRelease
- kuberture: external-dns public/internal hostname isolation
- e2e-install: tenant-name validation must reject (not warn); the version
ConfigMap must be gone after delete
- build-matrix: talos/installer excluded from the parallel matrix
- capture-dataplane: kube-dns and pending rows are filtered out
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
cozytest.sh — the runner for every bats file here — is not real bats: it
never invokes setup()/teardown(). So these hooks were dead. Cleanup ran
only via each test's success-path inline delete, and a failing test
leaked its resources into the shared tenant-test namespace (and, for
kuberture, cluster-scoped RBAC that persists across sandbox re-runs).
Wire the cleanup to cozy_cleanup(), the hook cozytest does run at suite
exit and on the first failing test:
- external-dns: dead teardown() -> cozy_cleanup() deleting all ExternalDNS
CRs. ExternalDNS is a stateless Deployment with the inmemory provider,
so a plain delete fully reclaims it — no drain needed.
- kuberture: dead teardown() -> cozy_cleanup() routed through the existing
_cleanup(), so the probe Pods, SA, and cluster-scoped ClusterRole and
ClusterRoleBinding are reclaimed on the failure path too.
build-matrix_test: drop the dead setup() entirely — it only cd'd to the
repo root, which cozytest already runs from, so the relative
hack/build-matrix.sh calls resolve without it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium leak healer escalates an unhealed leaked-IP reuse to a Tier-2
cilium-agent restart. For one variant that restart is both futile and
harmful: when the disputed IP is the node's per-node Cilium ingress IP
(CiliumNode.spec.ingress.ipv4), held by the reserved:ingress endpoint
(allocated from the node pod CIDR whenever Gateway API / Envoy is on),
restarting the agent cannot clear it -- the agent re-derives the same
ingress IP from the CiliumNode spec on every boot -- and can re-trigger
the very race that caused the wedge: the restart rebuilds the host-scope
(kubernetes) IPAM bitmap empty, then re-reserves the ingress IP via a
path decoupled from pod-endpoint restore, so a CNI ADD in that window can
be handed the ingress IP again. This is an upstream cilium bug in
host-scope IPAM with Gateway API; no released cilium fixes it as of
v1.19.5.
Detect that variant by an exact match of the wedged IP against the node's
CiliumNode.spec.ingress.ipv4 and skip the Tier-2 restart -- surface the
case and reschedule the wedged pod best-effort instead. The match is
gated so an empty ingress IP (CiliumNode absent or RBAC denied) never
matches, so a false positive can never suppress a restart that would
clear a real in-memory leak. Add a CiliumNode read to the healer RBAC and
a unit test pinning the exact-match-and-non-empty contract.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The current-main overlay skipped only rebuilt units (plus talos/installer),
so for a package the PR edited but did not rebuild — an upstream image bump
in a non-build-unit package such as keycloak or cert-manager — the committed
ref differed from the main artifact on image-reference lines only, the drift
check passed, and the overlay reverted the PR's edit back to current-main.
e2e and the installer artifact then ran against the stale image.
Pass the PR's edited package dirs (derived from the plan job's git diff) into
the overlay skip set so a PR's own image-ref edits win over the artifact. Add
tests pinning the preserve behaviour and covering the --image= arg form and
the file-absent-in-PR-tree branch.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Conditional PR builds (the per-package matrix in pull-requests.yaml) only
rebuild packages whose dir changed. Every other package falls back to the image
refs committed in the tree — the last release's digests (currently v1.5.0),
which go stale as main advances. So e2e and the installer's packages artifact
ran current-main configs against last-release images for everything outside the
PR's build matrix, and a PR installed via the cozy-installer chart was an
incoherent mix. This is the systemic gap in issue #3143 (its concrete bite: the
merged objectstorage-controller requeue fix#3034 never ran in e2e, flaking the
harbor BucketClaim check).
In finalize, pull the cozystack-packages:main OCI artifact that build-main
already publishes (the whole current-main packages tree, refs digest-pinned to
the images build-main pushed) and walk every ref-bearing file in it
(values.yaml + images/*.tag), overlaying the repo's copy. This is driven by the
artifact, not the build matrix, so it covers the ENTIRE first-party tree —
apps/system/core AND extra/, library/ — and a merged fix to any first-party
image (e.g. the objectstorage-sidecar ref under extra/seaweedfs) takes effect.
For a file build-main did not rebuild the artifact copy is byte-identical and
is skipped. Skipped explicitly: units the PR itself rebuilt (their pr-<N>-<sha>
refs win), packages/core/talos and packages/core/installer (owned by their
dedicated build-talos / finalize-installer jobs), and vendored charts/ subtrees.
The copy is self-validating: a file is overlaid only when every differing line
is image-reference-bearing (a full @sha256: ref, or a split
image/repository/registry/tag/digest key, or a --…-image= arg); otherwise it
keeps its committed ref. A missing artifact is a no-op (prior behaviour) and the
step is best-effort so it never blocks a PR.
No build-main change is needed: cozystack-packages:main is already published by
`make build`. Runs before the installer build so the flux-pushed packages
artifact and the pr.patch e2e applies both reflect the overlay.
Result: a PR is tested and installed as current-main + its own changes, coherent
across config and image, instead of current-main configs running on last-release
images.
Covered by hack/overlay-main-images_test.bats (run by `make unit-tests` in the
checks job): single-line and split-form ref overlay, extra/* coverage,
skip-rebuilt, talos/installer skip, vendored-charts prune, drift
self-validation, and missing-artifact no-op.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil