The master's /cluster/status:9333 handler calls topo.Leader(), which spins
an exponential backoff with MaxElapsedTime=20s before returning 200 whenever
the node has no leader in its view — e.g. a master that was briefly the initial
raft leader during bootstrap and then was demoted. The chart's master liveness
and readiness probes both httpGet /cluster/status with timeoutSeconds: 10, which
is shorter than that worst-case backoff. Such a master is then liveness-killed
roughly every 137s into a permanent CrashLoop, the master StatefulSet never
reaches full readiness, and the helm --wait install times out.
Raise both master probe timeouts to 30s so each probe outlasts the 20s backoff
(the volume component already uses 30s). The override sets only timeoutSeconds
and relies on Helm's deep-merge to keep the rest of the subchart probe defaults.
Both probes must be raised: leaving readiness at 10s would keep the master from
becoming Ready and still stall the StatefulSet.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Raises OpenSSF Scorecard **Pinned-Dependencies** (currently **1/10**) by
having Renovate manage and pin container base images and GitHub Actions
to immutable digests.
[PR #2849](https://github.com/cozystack/cozystack/pull/2849) SHA-pinned
actions in 13 workflows, but the Scorecard check also scores **container
base images** — 58 unpinned `FROM` lines across **27 distinct images** —
which dominate the score and keep it at 1/10. #2849 also left
`build-main.yaml` and `release-e2e.yaml` actions unpinned, and nothing
in the Renovate config kept the newly-added action SHAs updated (they
were set to rot).
### Changes
- **`.github/renovate.json`**
- Add `dockerfile` and `github-actions` to `enabledManagers` (was
`gomod`-only).
- `pinDigests: true` for both managers — Renovate opens the initial "pin
to `@sha256:`" PR (including the two workflows #2849 missed) and keeps
digests fresh thereafter.
- Batch the initial pin + all digest-only refreshes into a **single
weekly PR** (`matchUpdateTypes: ["digest", "pinDigest"]`, scheduled
Monday). Frequently-rebuilt bases (alpine, debian, golang, distroless)
would otherwise trickle in one rebuild-triggering PR each and load the
shared CI runner. Real tag bumps (major/minor/patch) are intentionally
excluded from the group so they get individual review.
- Existing `do-not-merge/hold` label + `automerge: false` are unchanged,
so every PR still requires a manual merge.
- **`packages/system/bucket/images/s3manager/Dockerfile`** — pin
floating tags to concrete versions so digest tracking doesn't churn
endlessly: `golang:1` → `golang:1.26`, `alpine:latest` → `alpine:3.24`.
### Notes
- The bulk digest pin of the 27 images is left to Renovate's first run
(one grouped PR) rather than hand-edited here, keeping this PR to the
config + tag-hygiene prerequisites.
- Expected effect once the follow-up pin PR merges: Pinned-Dependencies
**1 → ~7–8**. The remaining `downloadThenRun` / pip / npm items cap it
below 10.
### Release note
```release-note
ci(renovate): enable Renovate digest pinning for Dockerfile base images and GitHub Actions to improve the OpenSSF Scorecard Pinned-Dependencies score
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved automated dependency management for container images and
GitHub Actions, including clearer update grouping and digest pinning.
* Updated container base images to newer, fixed versions for more
predictable builds and deployments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The TenantNamespace registry helpers (`filterAccessible`,
`hasAccessToNamespace`, `hasAccessToNamespaceForUser`) returned plain
`fmt.Errorf` values, which the API server coerces into a generic `500
InternalError` regardless of the actual failure. This PR types the error
paths so clients get accurate status codes: a request without user
identity in context now returns `401 Unauthorized`, and a RoleBinding
list failure returns `500 InternalError` with a properly structured
`Status` body that preserves the underlying cause.
Additionally, `Get` now repackages a NotFound from the backing Namespace
lookup under the `tenantnamespaces` GroupResource, so the `Status`
details no longer leak the backing `namespaces` resource identity.
The `Watch` goroutine error path was audited as part of this change:
access-check failures there are logged and the event is skipped, so no
untyped error escapes to the client from that path. No changes were
needed.
All changed error paths are covered by new tests (helper-level plus
end-to-end through `Get` and `List`), using an interceptor-based fake
client that fails List only for RoleBindings.
Closes#2521
### Screenshots
No UI changes.
### Release note
```release-note
refactor(api): TenantNamespace API now returns 401 Unauthorized for requests without user identity and a structured 500 InternalError for RBAC lookup failures, instead of a generic 500
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved REST error handling to return standard Kubernetes API error
types, including proper scoping for missing tenant-namespace resources,
unauthorized requests when no user identity is present, and internal
failures when RoleBindings can’t be listed.
* **Tests**
* Expanded typed-error test coverage for access checks and listing
paths, including scenarios for missing user context, RoleBinding list
failures, and NotFound responses that surface the expected resource
details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Two fixes to the tenant chart's `pre-delete` cleanup hook (the Job that
deletes a tenant's HelmReleases when the tenant is removed).
**1. Grant the `watch` verb so the hook stops hanging.** The hook runs
`kubectl delete helmreleases ... --wait=true`. Since Kubernetes 1.25
that flag opens a watch on the target resources to block until
finalization. The hook's Role granted only `get`, `list`, `delete`, so
the watch was rejected with HTTP 403, `kubectl delete` hung
indefinitely, the Job never finished, and tenant deletion stalled on the
pre-delete hook. Adding `watch` brings the Role in line with the
`kubernetes` app chart's cleanup Role, which already grants it for the
same `--wait` deletes.
**2. Replace the unmaintained image.** The Job ran kubectl from
`bitnami/kubectl:latest`. The bitnami Docker Hub catalog has moved to a
legacy/sunset state, so that pull is at risk of breaking — which would
stop the hook from starting at all — and `:latest` leaves the kubectl
version unpinned. It now uses `docker.io/clastix/kubectl:v1.32`, the
version-pinned image already used by every other first-party
cleanup/delete hook in the repo (kubernetes, keycloak-configure,
dashboard, mariadb). That image is Alpine-based and ships only
`/bin/sh`, so the hook script (pure POSIX) now invokes `/bin/sh` instead
of `/bin/bash`.
A helm-unittest suite pins all three invariants (watch verb, image,
shell) so none can regress.
### Screenshots
N/A — no UI impact.
### Release note
```release-note
fix(tenant): grant the watch verb to the cleanup Role and move it to a maintained kubectl image so tenant deletion no longer hangs on the pre-delete hook
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Improvements**
* Expanded pre-delete cleanup RBAC permissions to include watching
HelmRelease resources.
* Hardened the cleanup Job runtime security settings for stronger
container confinement.
* Pinned the cleanup Job to a specific kubectl image version and
switched the command interpreter for safer execution.
* **Tests**
* Added/extended automated validation for the pre-delete cleanup hook,
covering RBAC rules, kubectl image pinning, command usage, and
securityContext hardening.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Updates kubevirt-operator from v1.8.2 to v1.8.4. This backports the VMI
checksum status-field validation fix (uint32 range) that Kubernetes 1.36
requires — without it, strict CRD numeric-format validation leaves VMIs
stuck in `Scheduled`. The release also carries assorted bug fixes and
CVE remediations. The v1.7.x line is skipped (live-migration regression,
kubevirt/kubevirt#16386).
The KubeVirt custom resource is unchanged — upstream ships a
byte-identical CR across these versions.
Verified on a dev cluster: all virt-* components roll out to v1.8.4 and
become ready.
Closes#2769
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
fix(kubevirt): update KubeVirt to v1.8.4; fixes VMIs stuck in Scheduled on Kubernetes 1.36 (strict CRD numeric-format validation of VMI checksum status fields)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the KubeVirt operator version from v1.8.2 to v1.8.4, including
the corresponding deployment configuration and operator
image/environment version bumps.
* Adjusted the operator update process to download the manifest using
v1.8.4 while skipping v1.7.x.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Completes the Go-toolchain modernization for first-party images — the
build-time counterpart to #2852, which raised the root and `api/`
modules to Go 1.26.4.
1. **First-party image Go modules → go 1.26.4** (`token-proxy`,
`kube-ovn-webhook`, `kubevirt-csi-driver`): raise the `go` directive and
refresh dependencies. `kube-ovn-webhook` moves `k8s.io/*` 0.26 → 0.36.2
(only long-stable admission/core API is used); `kubevirt-csi-driver`
keeps its existing replace-based k8s alignment at 0.36.0 (main's
refactor owns that module's pin) and raises the `go` directive, with the
`golang.org/x/net` 0.56.0 security bump — which clears the reachable
GO-2026-5026 — reapplied. Adds two `.gitignore` files so the
locally-built binaries can't be committed.
2. **Go builder images → `golang:1.26`** across 8 images still on
EOL/behind toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25):
`cluster-autoscaler`, `kubevirt-cloud-provider`, `redis-operator`,
`multus-cni`, `keycloak-operator`, `objectstorage`, `kilo`,
`linstor-csi`. The builder image tag — not the go.mod minimum — sets the
standard-library version baked into the binary, so these still shipped
unpatched stdlib on a plain rebuild.
Verified: `GOOS=linux go build`/`go vet` pass for all three modules and
`go mod tidy` is a no-op; the three biggest builder jumps
(redis-operator, kubevirt-cloud-provider, cluster-autoscaler — all
formerly Go 1.20.x/1.23.x) were verified with a local `docker build`;
the remaining five (1.24/1.25 → 1.26) are covered by CI image builds.
### Release note
```release-note
chore(platform): raise the Go toolchain to 1.26 across first-party images to clear standard-library security advisories
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated several container build environments to a newer Go release for
consistency across Kubernetes-related components.
* Refreshed module and dependency versions in a few services to keep
builds current.
* Expanded ignore rules for generated image artifacts so build outputs
aren’t tracked.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Every migration that stamps the `cozystack-version` ConfigMap must carry
the `platform.cozystack.io/no-delete=true` label so the
`cozystack-no-delete-guardrail` ValidatingAdmissionPolicy guards it. The
chart template renders the labeled ConfigMap only on first install, so
each upgrade-time stamp has to re-emit the label by hand — migration 43
had to inline-copy migration 42's heredoc to avoid a label-less apply by
the same field manager stripping it. The bug class was patched
per-migration, not eliminated.
This extracts the stamp into a shared, sourced helper
(`render_cozystack_version_manifest` + `stamp_cozystack_version`) under
`images/migrations/migrations/lib/`, and routes migrations 42, 43, 44
and the `run-migrations.sh` bootstrap path through it. The labeled
manifest now has a single definition, so future migrations cannot drift
back to a label-less apply.
The upgrade path is behavior-preserving: the rendered manifest is
byte-identical to the previous heredocs (pinned by a golden unit test).
The bootstrap path in `run-migrations.sh` previously created the
ConfigMap without the label — it now carries it like every
migration-driven stamp.
Adds `hack/cozystack-version-stamp.bats` (8 tests) covering the helper:
label presence, ConfigMap name, quoted-string version type, namespace
default and override, required-argument guard, and the byte-exact golden
manifest.
Closes#2781
### Release note
```release-note
fix(platform): always stamp the platform.cozystack.io/no-delete label on the cozystack-version ConfigMap, including the migration bootstrap path, so deletion protection cannot be silently dropped
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Tests**
* Expanded validation to confirm the version-stamp manifest is parseable
YAML, includes the required no-delete label, targets the correct
`cozystack-version` ConfigMap, and quotes the version value correctly.
* Added coverage for default and overridden namespaces, argument
validation failures, and a golden “byte-for-byte” render check.
* Added a guardrail test to prevent bypassing the shared stamping logic
by direct `kubectl` ConfigMap stamping.
* **Refactor**
* Centralized `cozystack-version` rendering and stamping into a shared
helper and updated migrations and the migrations runner to use it
consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Rebuild the kamaji image and force the fixed versions of three Go
modules whose advisories are compiled into the manager binary. Two build
changes:
- Advance the build source from `26.3.5-edge` to `26.3.6-edge`,
recompiling on the current `golang:1.26` toolchain (1.26.4) to close the
Go stdlib advisories.
- After applying the source patches and before `go build`, run `go get
golang.org/x/net@v0.55.0 golang.org/x/crypto@v0.52.0
go.opentelemetry.io/otel/sdk@v1.43.0 && go mod tidy` to pull the patched
module versions into the build.
## Why
The pinned image is flagged for a set of CVEs that fall into three
groups:
- 16 OS-package findings (libpython, libexpat, gnutls, krb5, sqlite,
curl, libgcrypt, libcap, nghttp2). The base is
`gcr.io/distroless/static:nonroot`, which ships no OS packages, so these
do not apply to the image — false positives.
- 1 `stdlib` finding: the binary was compiled with a go1.26 patch older
than 1.26.4. The `26.3.6-edge` rebuild on the current toolchain closes
it (crypto/x509 CVE-2026-27145, mime CVE-2026-42504, net/textproto
CVE-2026-42507).
- Go-module advisories that, unlike the OS-package noise, are genuinely
present in the compiled binary. `26.3.6-edge` pins `golang.org/x/net
v0.51.0`, `golang.org/x/crypto v0.49.0` and
`go.opentelemetry.io/otel/sdk v1.41.0`, all carrying published
advisories:
- `golang.org/x/net` < 0.55.0: CVE-2026-42506, CVE-2026-42502,
CVE-2026-39821, CVE-2026-25680, CVE-2026-25681, CVE-2026-27136 (plus
CVE-2026-33814, fixed at 0.53.0). Target 0.55.0.
- `golang.org/x/crypto` < 0.52.0: 13 advisories in
`golang.org/x/crypto/ssh` (e.g. CVE-2026-39827, CVE-2026-46595,
CVE-2026-46598). Target 0.52.0.
- `go.opentelemetry.io/otel/sdk` < 1.43.0: CVE-2026-39883
(GHSA-hfvc-g4fc-pqhx), a PATH-hijack on BSD/Solaris only; the affected
code path does not run on the Linux distroless target, but the bump
clears the finding at no runtime cost. Target 1.43.0.
A source-version bump cannot close the module advisories: no kamaji edge
tag up to `26.6.4-edge` ships the fixed module versions (it still pins
x/net 0.53.0, x/crypto 0.51.0, otel/sdk 1.41.0), and the tags that
advance the k8s line to 1.36 regress tenant control-plane RBAC. Forcing
the modules during the build is the minimal correct lever, and the
overlay already triggers a fresh compile, so it subsumes the stdlib
rebuild.
## Verification
Built the image source end to end: extracted `26.3.6-edge`, applied the
patch, ran the overlay (`go get` + `go mod tidy`, clean), and compiled
with `CGO_ENABLED=0 go build -a -o kamaji main.go`. The build succeeds
and `go version -m` on the binary embeds `golang.org/x/net v0.55.0`,
`golang.org/x/crypto v0.52.0` and the `go.opentelemetry.io/otel` set at
v1.43.0. The source has no `vendor/` directory, so `go get` + `go mod
tidy` is sufficient. Advisory IDs and fixed boundaries verified against
the OSV database.
After this change the shipped binary carries no Go advisory with an
available fix; the only residual scanner entries are in
`k8s.io/kubernetes` with no upstream fix, which no version bump can
close.
Closes#2885
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the Kamaji image version to a newer patch release.
* Refreshed several bundled Go dependencies to newer pinned versions
before building.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The mariadb-operator hardcodes the mysqld-exporter liveness and readiness
probe as an httpGet on port 9104, which kubelet dials over the pod's IPv4
address. The exporter's default [::] (IPv6 wildcard) listener intermittently
refuses the IPv4 connect on single-stack-IPv4 data planes, so readiness fails
shortly after start, the pod flaps to NotReady, and the metrics Service keeps
the pod in notReadyAddresses — its endpoint never populates.
Pin the exporter to the IPv4 wildcard via --web.listen-address=0.0.0.0:9104 so
the kubelet probe connects deterministically. The operator appends user args to
its defaults and does not itself set --web.listen-address, so there is no
duplicate-flag conflict.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Fixes two failure modes in the COSI control plane built from upstream
container-object-storage-interface v0.2.2, both carried as image
patches.
**BucketClaim never converges to ready (#3032).** The controller does
not watch Bucket objects, and the generic controller drops no-op resync
deltas, so a BucketClaim whose backend Bucket flips to ready *after* the
first reconcile stayed `bucketReady=false` indefinitely — BucketAccess
was never granted and the consuming app (bucket, harbor) got no
credentials. The existing re-read of the live Bucket only ran during a
reconcile that nothing re-triggered. The controller now returns an error
while the Bucket is not ready, forcing a rate-limited requeue that
re-reads the Bucket until its readiness propagates, so the claim
converges deterministically instead of racing the async sidecar.
**Slow provisioner failover on pod recreate (part of #2429).** The
controller and the per-driver sidecar run single-replica, so leader
election adds no availability — only failover latency. The upstream 150s
lease made a recreated provisioner pod stall reconciliation for up to
~150s. The lease is lowered to the standard Kubernetes values
(15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket
CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver
v0.3.1 self-heal; the silent "Bucket ready with no backend bucket"
finding on that issue is a separate driver bug and is left tracked
there.
Coverage: a unit test in the patched controller pins the
requeue/propagation contract (fails without the requeue), and the
bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound
instead of tolerating the lag, with scoped diagnostics on the failure
path.
Closes#3032
Refs #2429
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved object storage bucket readiness propagation so bucket claims
now track the backend bucket’s actual `bucketReady` status and requeue
when readiness isn’t yet observed.
* Reduced and made readiness waits more reliable in end-to-end checks,
with explicit match conditions and clearer debug output (including claim
YAML, backend bucket list, and controller pod status).
* Shortened leader-election timing to speed up recovery after pod
restarts.
* **Chores**
* Updated automated PR labeling to map the object storage controller
scope to the storage area label.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Make it unambiguous where each kind of contribution lands:
| Contribution | Destination |
|---|---|
| Bug | issue here (`kind/bug`) |
| Concrete, scoped feature | issue here (`kind/feature`) |
| Question / early idea | GitHub Discussions |
| Cross-cutting / architectural change | design proposal in
cozystack/community |
| Governance / process / community | cozystack/community |
## Changes
- Add `.github/ISSUE_TEMPLATE/config.yml` — a chooser
(`blank_issues_enabled: false`) linking to Discussions, the community
design-proposal process, community governance issues, and chat.
- Add `.github/ISSUE_TEMPLATE/feature_request.md` (`kind/feature`) with
an up-front gate (question → Discussion; cross-cutting → design
proposal) and a scope checklist.
- `CONTRIBUTING.md`: add a "Where do I file?" routing table.
- `README.md`: feature requests now point at issues instead of a
Discussions category, reconciling a contradiction with the actively-used
issue tracker.
## Notes
- `blank_issues_enabled: false` removes the blank-issue escape hatch to
keep triage clean — easy to flip if you'd rather keep it.
- Pairs with a companion PR in cozystack/community (routing table +
dead-link fix).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added “Where do I file?” guidance to help route bugs, scoped feature
requests, questions/early ideas, cross-cutting design proposals, and
community/process topics.
* Updated the README “Contributions” section with clearer links and
pointers to the right place for each request type.
* **New Features**
* Added a structured “Feature request” issue template to collect
consistent, scoped details.
* Disabled blank issue submissions and provided curated entry points for
common support and discussion paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
`packages/apps/kafka` rendered the Strimzi `Kafka` CR with
`entityOperator.topicOperator: {}` / `userOperator: {}` and no
`resources`. Tenant namespaces ship a `LimitRange` that defaults
containers to `memory: 128Mi`, so the topic-operator and user-operator
JVM containers (Kafka 3.9.1 + fabric8/vertx) are **OOMKilled (exit
137)** on startup. The `<cluster>-entity-operator` Deployment never
becomes Ready (CrashLoopBackOff), so:
- `KafkaTopic` / `KafkaUser` CRs never reconcile — topic creation
effectively fails;
- the cluster-operator logs `Exceeded timeout of 300000ms while waiting
for Deployment ...-entity-operator ... to be ready`;
- on teardown, `KafkaTopic`s get stuck permanently with the
`strimzi.io/topic-operator` finalizer (no operator left to clear them),
blocking namespace deletion.
The Kafka broker + ZooKeeper come up fine — only the entity-operator
dies. The `apps.cozystack.io/Kafka` schema exposes no field for
entity-operator resources, so there is no user-side workaround.
## Fix
Set explicit `resources` (requests `256Mi`/`100m`, limit `512Mi` memory)
on `entityOperator.topicOperator` and `userOperator` so they no longer
inherit the 128Mi namespace default. Adds a `helm-unittest` assertion
for the rendered limits.
## Testing
Reproduced on a clean cluster (Cozystack v1.4.3 and v1.5.0-rc.2; Strimzi
0.45.1, Kafka 3.9.1): both entity-operator containers `OOMKilled` at
128Mi → CrashLoopBackOff → `KafkaTopic` never `READY`. After bumping to
512Mi the entity-operator rolls cleanly (both containers Ready, 0
restarts) and the `KafkaTopic` reconciles to `READY=True`.
### Release note
```release-note
fix(kafka): give entity-operator (topic/user-operator) explicit resources so they no longer inherit the 128Mi tenant LimitRange default and OOMKill, which left KafkaTopics/KafkaUsers unreconciled.
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Kafka topic and user operators now use explicit CPU and memory
requests and limits, avoiding reliance on namespace-level defaults.
* **Tests**
* Added regression coverage to verify the Kafka entity operator’s topic
and user operator memory limits are explicitly set (512Mi).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The kafka e2e test pinned both the kafka and zookeeper components to the
legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the
Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup
(exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka
CR and its HelmRelease never go Ready, and the 5m readiness wait times
out. It surfaced as a flake because 128Mi sits right at the JVM-start
threshold — occasionally a broker passes readiness before its RSS crosses
the cgroup limit.
Raise both components to "c1.small" (1 CPU / 1Gi), the chart default,
which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so
1Gi across the four pods is well within budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).
Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.
RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ovn-controller runs inside the ovs-ovn DaemonSet's openvswitch container,
sharing it with ovsdb-server and ovs-vswitchd. It is CPU-bound while
programming OpenFlow for newly created logical ports. Under a burst install
(many pods created at once) it competes for CPU in that single container and
CFS-throttles against the 2-core limit, lengthening flow-programming lag that
leaves freshly created pods unreachable from their node until the flows land.
Raise the ovs-ovn CPU limit from 2 to 4 cores to give ovn-controller headroom
to program flows promptly under burst, and lift the CPU request from 10m to
100m so it keeps a guaranteed share under node CPU contention. Memory is left
unchanged. Add a helm-unittest guard pinning the new CPU limit and request.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fluent-bit DaemonSet crash-loops on a loaded install, timing out its
HelmRelease and cascading into velero and backupstrategy-controller
DependencyNotReady failures. The subchart default liveness/readiness probes
(initialDelaySeconds 0, timeoutSeconds 1, failureThreshold 3) target the HTTP
server on :2020, but that server does not bind until AFTER the kubernetes
filter's 'testing connectivity with API server' step. Under a concurrent
install the kube-API can be unreachable for ~10s, delaying the :2020 bind,
while the single-threaded HTTP server can miss a 1s timeout when the engine is
busy. The probes then fire into a connection-refused port and liveness-kill
fluent-bit into a restart loop.
Give both probes load headroom (initialDelaySeconds 15, timeoutSeconds 5,
failureThreshold 6) so the API-gated late bind and busy-engine blips are
tolerated. The httpGet target is unchanged (path / on port http = :2020). Add
a helm-unittest pinning the relaxed timing and the preserved target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
While the backend Bucket is not ready the controller returns an error to
force a requeue, but that error is only logged at klog.V(3) and is
suppressed at default verbosity, so an operator cannot see why the claim
is stuck. Record a WaitingForBucket Warning event on the BucketClaim on
each requeue (event aggregation folds the repetition) and still return
the error so the requeue happens as before. Pin the requeue contract in
the test by asserting the error carries the stable "not ready yet"
substring, not merely that it is non-nil.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Pull Request Build job authenticates only to the push registry (OCIR),
so every base image (golang, ubuntu, alpine, nginx, ...) is pulled from
Docker Hub anonymously. Those pulls intermittently fail at
auth.docker.io/token under Docker Hub's anonymous-pull rate limit, leaving
the base image unresolved and failing make build.
Build on a docker-container buildx builder whose buildkitd routes docker.io
through mirror.gcr.io, Google's public pull-through cache for Docker Hub: no
auth, far higher limits, transparent fall-through to Docker Hub on a miss.
This reuses the builder pattern already proven in build-main.yaml (same
make build, same DOCKER_CONFIG auth forwarding); only the committed
hack/buildkitd.toml mirror config is new.
Fail-safe: if mirror.gcr.io is unreachable, BuildKit falls back to
registry-1.docker.io and the build still succeeds, so this strictly reduces
build failures without adding a new hard dependency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Under node CPU contention the linstor-controller JVM cannot answer its GET /health liveness probe within the piraeus-operator defaults (timeoutSeconds 1, failureThreshold 3). The kubelet then liveness-kills the container; the resulting crash-loop leaves the linstor-controller Service with no ready endpoints, linstor-csi gets EPERM, and RWX-Filesystem DataVolumes never bind, breaking tenant provisioning.
The operator-generated controller pod sets no resources (no CPU request, hence no scheduling/CPU guarantee) and ships the tight default probe. Add CPU/memory requests and relax the liveness timing via the controller podTemplate, which piraeus-operator merges into the Deployment as a strategic merge patch. The probe override carries only the timing fields; the httpGet target (GET /health on the api port) is left to the operator to avoid pinning a wrong path or scheme. Requests-only (no limits) keeps the JVM from being throttled or OOM-killed. The knobs are exposed under controller.* for tuning to observed RSS.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The operator Deployment ships a single replica but always passes
--leader-elect to the manager. On one replica leader election adds no HA
value: a transient apiserver/etcd lease-renewal blip makes
controller-runtime self-terminate (os.Exit on "leader election lost"),
which crashloops the manager and surfaces at install time as a
Deployment stuck in Failed.
Gate --leader-elect on the replica count via a manager.replicaCount
value (default 1). The single-replica install now rides out
control-plane latency instead of crashlooping, and scaling to more than
one replica re-enables leader election automatically for genuine HA.
Also set the Deployment strategy to Recreate on a single replica: with
leader election off, a default RollingUpdate would briefly run two
managers (maxSurge) during an operator upgrade, both reconciling without
lease coordination. Recreate closes that window at zero availability
cost, while installs scaled past one replica keep RollingUpdate.
The change to the vendored opensearch-operator 2.8.0 chart is carried in
patches/leaderElection.diff and re-applied by 'make update' so it
survives a chart refresh.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.
Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The reconciled, hasListenIPsForAllPods, fullReplication and healthy gates
polled status fields with a 60s ceiling, while the strictly-earlier
health.available gate had 300s. The operator stamps generations.reconciled
only after every process holds its role for minimumUptimeSecondsForBounce
(60s), making reconciled and the health fields that depend on it the last
status fields to converge. The short ceiling closed the gate seconds before
the field landed on a healthy, fully-reconciled cluster, causing intermittent
jsonpath poll timeouts.
Raise those four gates to 300s to match the health.available budget. The waits
stay event-driven; only the ceilings widen.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The "Bootstrap Talos cluster" step times out the etcd-members health gate
at 180s. A 3-node etcd converges via Talos's serialized learner promotion
(one learner admitted at a time), so on a loaded ephemeral runner this
deterministic path can approach ~3m wall clock and photo-finish against the
180s ceiling — failing a cluster that is in fact healthy. The wait is
already event-driven; only the ceiling was too tight. Widen it to 300s.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The kamaji source up to the 26.6.4-edge tag pins Go modules that carry
known advisories compiled into the manager binary: golang.org/x/net
below 0.55.0, golang.org/x/crypto below 0.52.0 and
go.opentelemetry.io/otel/sdk below 1.43.0. Unlike the distroless
OS-package findings, these advisories ship inside the Go binary and are
reported against the deployed image.
Force the fixed module versions during the build, after applying the
source patches and before compiling, so the binary embeds them. No edge
tag ships the fixes, and the tags that advance these modules also bring
the k8s 1.36 kubeadm change that regresses tenant control-plane RBAC, so
a source version bump is not a viable alternative.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The objectstorage-controller package ships the COSI control plane; its commits
use the objectstorage-controller scope, which had no area mapping and fell back
to area/uncategorized. Map it to area/storage alongside the other COSI/storage
scopes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.
Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The shipped kamaji binary was compiled with go1.26.3, which trivy flags
for the standard-library CVEs fixed in go1.26.4: crypto/x509
CVE-2026-27145, mime CVE-2026-42504, and net/textproto CVE-2026-42507.
The image base is distroless/static, so the remaining OS-package CVEs
from the scan do not apply to it.
Advance the build source from 26.3.5-edge to 26.3.6-edge, the next
patch on the same edge line (identical k8s.io/api v0.35.0 and
controller-runtime v0.22.4, staying clear of the k8s 1.36 line that
regresses tenant control-plane RBAC). The version change busts the
build-layer cache so the release rebuild compiles against the current
golang:1.26 toolchain, which resolves to 1.26.4 and closes the stdlib
CVEs.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The controller and the per-driver sidecar both run single-replica, so leader
election provides no high availability -- it only governs how quickly a freshly
recreated pod takes over. With the upstream 150s lease duration a recreated
provisioner pod stalls reconciliation for up to 150s while it waits out the
stale lease still held by the previous, now-deleted pod. Lower the lease timings
to the standard Kubernetes values (15s/10s/2s) so a recreated pod resumes
reconciliation in roughly 15s.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dynamic-provisioning path re-reads the live Bucket after create and copies
its readiness onto the BucketClaim, but only during a reconcile. Nothing
re-drives that reconcile afterwards: the controller does not watch Buckets, and
the generic controller drops no-op resync deltas (reflect.DeepEqual), so once
the BucketClaim stops changing the listener is never re-invoked. The Bucket's
false->true transition, performed asynchronously by the sidecar on a different
object, is therefore reflected onto the claim only if the Bucket happened to be
ready during that single reconcile -- a race the sidecar usually loses, leaving
bucketReady frozen at false and BucketAccess never granted.
Return an error while the backend Bucket is not ready so the work queue
requeues the claim with exponential backoff; each retry re-reads the live
Bucket until its readiness has been propagated, converging deterministically
instead of depending on a resync that never fires.
Add a unit test pinning the contract (requeue while not ready, propagate true
once ready) and adjust the existing status-update conflict-retry test, which
pinned the old never-requeue behavior.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Aleksei Sviridkin