The master's /cluster/status:9333 handler calls topo.Leader(), which spins
an exponential backoff with MaxElapsedTime=20s before returning 200 whenever
the node has no leader in its view — e.g. a master that was briefly the initial
raft leader during bootstrap and then was demoted. The chart's master liveness
and readiness probes both httpGet /cluster/status with timeoutSeconds: 10, which
is shorter than that worst-case backoff. Such a master is then liveness-killed
roughly every 137s into a permanent CrashLoop, the master StatefulSet never
reaches full readiness, and the helm --wait install times out.
Raise both master probe timeouts to 30s so each probe outlasts the 20s backoff
(the volume component already uses 30s). The override sets only timeoutSeconds
and relies on Helm's deep-merge to keep the rest of the subchart probe defaults.
Both probes must be raised: leaving readiness at 10s would keep the master from
becoming Ready and still stall the StatefulSet.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The admin Ingress always requested a per-host ACME certificate, ignoring
the operator wildcard mode that the login Ingress already respects. When
the admin route shares the public ingressClass, the operator default SSL
cert covers it too, so skip per-host ACME there. A separate admin ingress
keeps its own certificate, since that controller has no wildcard default.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
When ingress.adminHost is set, narrow the public (login) route to the
OIDC paths only (/realms, /resources, /.well-known) so the admin console
and Administration REST API are no longer served on the login hostname,
and serve them from a dedicated admin route instead. The admin route
attaches to the Gateway/ingressClass selected by _cluster.expose-ingress-admin
(defaults to expose-ingress), so it can be moved onto a separate, e.g.
private, Gateway. KC_HOSTNAME_ADMIN alone is not a security boundary;
the path and network separation here is what restricts access.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Expose a new opt-in publishing.ingressNameAdmin that is propagated to
_cluster.expose-ingress-admin. System components (currently Keycloak)
use it to attach their admin routes to a separate Gateway/ingressClass,
e.g. one bound to a private IP and unreachable from the public internet.
Empty by default, so admin endpoints keep sharing expose-ingress. The
private Gateway itself is provisioned out of band; this only selects
where the admin route attaches.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Add helm-unittest suite asserting KC_HOSTNAME_ADMIN env, the extra
Ingress rule/TLS entry, and the appended HTTPRoute hostname are emitted
only when ingress.adminHost is set. Wire up the chart 'test' target.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Add ingress.adminHost option to serve the Keycloak admin console and
admin REST API on a dedicated hostname, separate from the login/realm
hostname. When set, KC_HOSTNAME_ADMIN is configured and an additional
Ingress rule / HTTPRoute hostname routes the admin domain to the same
Keycloak service. Empty by default, keeping current behavior.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Make the Keycloak database connection target configurable so it can be routed
through an external endpoint (for example a pgwire proxy) without changing the
chart:
- db.host / db.port override the host/port otherwise read from the
keycloak-db-app secret;
- db.urlProperties sets KC_DB_URL_PROPERTIES (previously never set), e.g.
sslmode=verify-full&prepareThreshold=0&sslrootcert=...;
- extraVolumes / extraVolumeMounts allow mounting a CA for sslrootcert.
All new values default to empty, so the rendered StatefulSet is byte-for-byte
unchanged for existing installs.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Renovate skips indirect gomod deps by default, so transitive modules
(golang.org/x/crypto, x/net, x/sys) that carry OSV advisories never get
a PR — leaving the OpenSSF Scorecard Vulnerabilities check pinned at 0
even after #3019 bumped x/crypto in the root module only.
- enable indirect gomod updates, grouped into one rolling PR
- turn on osvVulnerabilityAlerts so vulnerable deps get dedicated,
mergeable security PRs (labeled `security`, not `do-not-merge/hold`)
Does not address GO-2025-3512 (kubevirt.io/csi-driver): no fixed version
exists upstream and Renovate cannot resolve the module, so the
Vulnerabilities check stays at 0 until that advisory is suppressed or
patched separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Raises OpenSSF Scorecard **Pinned-Dependencies** (currently **1/10**) by
having Renovate manage and pin container base images and GitHub Actions
to immutable digests.
[PR #2849](https://github.com/cozystack/cozystack/pull/2849) SHA-pinned
actions in 13 workflows, but the Scorecard check also scores **container
base images** — 58 unpinned `FROM` lines across **27 distinct images** —
which dominate the score and keep it at 1/10. #2849 also left
`build-main.yaml` and `release-e2e.yaml` actions unpinned, and nothing
in the Renovate config kept the newly-added action SHAs updated (they
were set to rot).
### Changes
- **`.github/renovate.json`**
- Add `dockerfile` and `github-actions` to `enabledManagers` (was
`gomod`-only).
- `pinDigests: true` for both managers — Renovate opens the initial "pin
to `@sha256:`" PR (including the two workflows #2849 missed) and keeps
digests fresh thereafter.
- Batch the initial pin + all digest-only refreshes into a **single
weekly PR** (`matchUpdateTypes: ["digest", "pinDigest"]`, scheduled
Monday). Frequently-rebuilt bases (alpine, debian, golang, distroless)
would otherwise trickle in one rebuild-triggering PR each and load the
shared CI runner. Real tag bumps (major/minor/patch) are intentionally
excluded from the group so they get individual review.
- Existing `do-not-merge/hold` label + `automerge: false` are unchanged,
so every PR still requires a manual merge.
- **`packages/system/bucket/images/s3manager/Dockerfile`** — pin
floating tags to concrete versions so digest tracking doesn't churn
endlessly: `golang:1` → `golang:1.26`, `alpine:latest` → `alpine:3.24`.
### Notes
- The bulk digest pin of the 27 images is left to Renovate's first run
(one grouped PR) rather than hand-edited here, keeping this PR to the
config + tag-hygiene prerequisites.
- Expected effect once the follow-up pin PR merges: Pinned-Dependencies
**1 → ~7–8**. The remaining `downloadThenRun` / pip / npm items cap it
below 10.
### Release note
```release-note
ci(renovate): enable Renovate digest pinning for Dockerfile base images and GitHub Actions to improve the OpenSSF Scorecard Pinned-Dependencies score
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved automated dependency management for container images and
GitHub Actions, including clearer update grouping and digest pinning.
* Updated container base images to newer, fixed versions for more
predictable builds and deployments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The TenantNamespace registry helpers (`filterAccessible`,
`hasAccessToNamespace`, `hasAccessToNamespaceForUser`) returned plain
`fmt.Errorf` values, which the API server coerces into a generic `500
InternalError` regardless of the actual failure. This PR types the error
paths so clients get accurate status codes: a request without user
identity in context now returns `401 Unauthorized`, and a RoleBinding
list failure returns `500 InternalError` with a properly structured
`Status` body that preserves the underlying cause.
Additionally, `Get` now repackages a NotFound from the backing Namespace
lookup under the `tenantnamespaces` GroupResource, so the `Status`
details no longer leak the backing `namespaces` resource identity.
The `Watch` goroutine error path was audited as part of this change:
access-check failures there are logged and the event is skipped, so no
untyped error escapes to the client from that path. No changes were
needed.
All changed error paths are covered by new tests (helper-level plus
end-to-end through `Get` and `List`), using an interceptor-based fake
client that fails List only for RoleBindings.
Closes#2521
### Screenshots
No UI changes.
### Release note
```release-note
refactor(api): TenantNamespace API now returns 401 Unauthorized for requests without user identity and a structured 500 InternalError for RBAC lookup failures, instead of a generic 500
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved REST error handling to return standard Kubernetes API error
types, including proper scoping for missing tenant-namespace resources,
unauthorized requests when no user identity is present, and internal
failures when RoleBindings can’t be listed.
* **Tests**
* Expanded typed-error test coverage for access checks and listing
paths, including scenarios for missing user context, RoleBinding list
failures, and NotFound responses that surface the expected resource
details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Two fixes to the tenant chart's `pre-delete` cleanup hook (the Job that
deletes a tenant's HelmReleases when the tenant is removed).
**1. Grant the `watch` verb so the hook stops hanging.** The hook runs
`kubectl delete helmreleases ... --wait=true`. Since Kubernetes 1.25
that flag opens a watch on the target resources to block until
finalization. The hook's Role granted only `get`, `list`, `delete`, so
the watch was rejected with HTTP 403, `kubectl delete` hung
indefinitely, the Job never finished, and tenant deletion stalled on the
pre-delete hook. Adding `watch` brings the Role in line with the
`kubernetes` app chart's cleanup Role, which already grants it for the
same `--wait` deletes.
**2. Replace the unmaintained image.** The Job ran kubectl from
`bitnami/kubectl:latest`. The bitnami Docker Hub catalog has moved to a
legacy/sunset state, so that pull is at risk of breaking — which would
stop the hook from starting at all — and `:latest` leaves the kubectl
version unpinned. It now uses `docker.io/clastix/kubectl:v1.32`, the
version-pinned image already used by every other first-party
cleanup/delete hook in the repo (kubernetes, keycloak-configure,
dashboard, mariadb). That image is Alpine-based and ships only
`/bin/sh`, so the hook script (pure POSIX) now invokes `/bin/sh` instead
of `/bin/bash`.
A helm-unittest suite pins all three invariants (watch verb, image,
shell) so none can regress.
### Screenshots
N/A — no UI impact.
### Release note
```release-note
fix(tenant): grant the watch verb to the cleanup Role and move it to a maintained kubectl image so tenant deletion no longer hangs on the pre-delete hook
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Improvements**
* Expanded pre-delete cleanup RBAC permissions to include watching
HelmRelease resources.
* Hardened the cleanup Job runtime security settings for stronger
container confinement.
* Pinned the cleanup Job to a specific kubectl image version and
switched the command interpreter for safer execution.
* **Tests**
* Added/extended automated validation for the pre-delete cleanup hook,
covering RBAC rules, kubectl image pinning, command usage, and
securityContext hardening.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Updates kubevirt-operator from v1.8.2 to v1.8.4. This backports the VMI
checksum status-field validation fix (uint32 range) that Kubernetes 1.36
requires — without it, strict CRD numeric-format validation leaves VMIs
stuck in `Scheduled`. The release also carries assorted bug fixes and
CVE remediations. The v1.7.x line is skipped (live-migration regression,
kubevirt/kubevirt#16386).
The KubeVirt custom resource is unchanged — upstream ships a
byte-identical CR across these versions.
Verified on a dev cluster: all virt-* components roll out to v1.8.4 and
become ready.
Closes#2769
### Screenshots
Not applicable — no UI changes.
### Release note
```release-note
fix(kubevirt): update KubeVirt to v1.8.4; fixes VMIs stuck in Scheduled on Kubernetes 1.36 (strict CRD numeric-format validation of VMI checksum status fields)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the KubeVirt operator version from v1.8.2 to v1.8.4, including
the corresponding deployment configuration and operator
image/environment version bumps.
* Adjusted the operator update process to download the manifest using
v1.8.4 while skipping v1.7.x.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Completes the Go-toolchain modernization for first-party images — the
build-time counterpart to #2852, which raised the root and `api/`
modules to Go 1.26.4.
1. **First-party image Go modules → go 1.26.4** (`token-proxy`,
`kube-ovn-webhook`, `kubevirt-csi-driver`): raise the `go` directive and
refresh dependencies. `kube-ovn-webhook` moves `k8s.io/*` 0.26 → 0.36.2
(only long-stable admission/core API is used); `kubevirt-csi-driver`
keeps its existing replace-based k8s alignment at 0.36.0 (main's
refactor owns that module's pin) and raises the `go` directive, with the
`golang.org/x/net` 0.56.0 security bump — which clears the reachable
GO-2026-5026 — reapplied. Adds two `.gitignore` files so the
locally-built binaries can't be committed.
2. **Go builder images → `golang:1.26`** across 8 images still on
EOL/behind toolchains (golang 1.20, 1.20.6, 1.23.4, 1.24, 1.25):
`cluster-autoscaler`, `kubevirt-cloud-provider`, `redis-operator`,
`multus-cni`, `keycloak-operator`, `objectstorage`, `kilo`,
`linstor-csi`. The builder image tag — not the go.mod minimum — sets the
standard-library version baked into the binary, so these still shipped
unpatched stdlib on a plain rebuild.
Verified: `GOOS=linux go build`/`go vet` pass for all three modules and
`go mod tidy` is a no-op; the three biggest builder jumps
(redis-operator, kubevirt-cloud-provider, cluster-autoscaler — all
formerly Go 1.20.x/1.23.x) were verified with a local `docker build`;
the remaining five (1.24/1.25 → 1.26) are covered by CI image builds.
### Release note
```release-note
chore(platform): raise the Go toolchain to 1.26 across first-party images to clear standard-library security advisories
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated several container build environments to a newer Go release for
consistency across Kubernetes-related components.
* Refreshed module and dependency versions in a few services to keep
builds current.
* Expanded ignore rules for generated image artifacts so build outputs
aren’t tracked.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Every migration that stamps the `cozystack-version` ConfigMap must carry
the `platform.cozystack.io/no-delete=true` label so the
`cozystack-no-delete-guardrail` ValidatingAdmissionPolicy guards it. The
chart template renders the labeled ConfigMap only on first install, so
each upgrade-time stamp has to re-emit the label by hand — migration 43
had to inline-copy migration 42's heredoc to avoid a label-less apply by
the same field manager stripping it. The bug class was patched
per-migration, not eliminated.
This extracts the stamp into a shared, sourced helper
(`render_cozystack_version_manifest` + `stamp_cozystack_version`) under
`images/migrations/migrations/lib/`, and routes migrations 42, 43, 44
and the `run-migrations.sh` bootstrap path through it. The labeled
manifest now has a single definition, so future migrations cannot drift
back to a label-less apply.
The upgrade path is behavior-preserving: the rendered manifest is
byte-identical to the previous heredocs (pinned by a golden unit test).
The bootstrap path in `run-migrations.sh` previously created the
ConfigMap without the label — it now carries it like every
migration-driven stamp.
Adds `hack/cozystack-version-stamp.bats` (8 tests) covering the helper:
label presence, ConfigMap name, quoted-string version type, namespace
default and override, required-argument guard, and the byte-exact golden
manifest.
Closes#2781
### Release note
```release-note
fix(platform): always stamp the platform.cozystack.io/no-delete label on the cozystack-version ConfigMap, including the migration bootstrap path, so deletion protection cannot be silently dropped
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Tests**
* Expanded validation to confirm the version-stamp manifest is parseable
YAML, includes the required no-delete label, targets the correct
`cozystack-version` ConfigMap, and quotes the version value correctly.
* Added coverage for default and overridden namespaces, argument
validation failures, and a golden “byte-for-byte” render check.
* Added a guardrail test to prevent bypassing the shared stamping logic
by direct `kubectl` ConfigMap stamping.
* **Refactor**
* Centralized `cozystack-version` rendering and stamping into a shared
helper and updated migrations and the migrations runner to use it
consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Rebuild the kamaji image and force the fixed versions of three Go
modules whose advisories are compiled into the manager binary. Two build
changes:
- Advance the build source from `26.3.5-edge` to `26.3.6-edge`,
recompiling on the current `golang:1.26` toolchain (1.26.4) to close the
Go stdlib advisories.
- After applying the source patches and before `go build`, run `go get
golang.org/x/net@v0.55.0 golang.org/x/crypto@v0.52.0
go.opentelemetry.io/otel/sdk@v1.43.0 && go mod tidy` to pull the patched
module versions into the build.
## Why
The pinned image is flagged for a set of CVEs that fall into three
groups:
- 16 OS-package findings (libpython, libexpat, gnutls, krb5, sqlite,
curl, libgcrypt, libcap, nghttp2). The base is
`gcr.io/distroless/static:nonroot`, which ships no OS packages, so these
do not apply to the image — false positives.
- 1 `stdlib` finding: the binary was compiled with a go1.26 patch older
than 1.26.4. The `26.3.6-edge` rebuild on the current toolchain closes
it (crypto/x509 CVE-2026-27145, mime CVE-2026-42504, net/textproto
CVE-2026-42507).
- Go-module advisories that, unlike the OS-package noise, are genuinely
present in the compiled binary. `26.3.6-edge` pins `golang.org/x/net
v0.51.0`, `golang.org/x/crypto v0.49.0` and
`go.opentelemetry.io/otel/sdk v1.41.0`, all carrying published
advisories:
- `golang.org/x/net` < 0.55.0: CVE-2026-42506, CVE-2026-42502,
CVE-2026-39821, CVE-2026-25680, CVE-2026-25681, CVE-2026-27136 (plus
CVE-2026-33814, fixed at 0.53.0). Target 0.55.0.
- `golang.org/x/crypto` < 0.52.0: 13 advisories in
`golang.org/x/crypto/ssh` (e.g. CVE-2026-39827, CVE-2026-46595,
CVE-2026-46598). Target 0.52.0.
- `go.opentelemetry.io/otel/sdk` < 1.43.0: CVE-2026-39883
(GHSA-hfvc-g4fc-pqhx), a PATH-hijack on BSD/Solaris only; the affected
code path does not run on the Linux distroless target, but the bump
clears the finding at no runtime cost. Target 1.43.0.
A source-version bump cannot close the module advisories: no kamaji edge
tag up to `26.6.4-edge` ships the fixed module versions (it still pins
x/net 0.53.0, x/crypto 0.51.0, otel/sdk 1.41.0), and the tags that
advance the k8s line to 1.36 regress tenant control-plane RBAC. Forcing
the modules during the build is the minimal correct lever, and the
overlay already triggers a fresh compile, so it subsumes the stdlib
rebuild.
## Verification
Built the image source end to end: extracted `26.3.6-edge`, applied the
patch, ran the overlay (`go get` + `go mod tidy`, clean), and compiled
with `CGO_ENABLED=0 go build -a -o kamaji main.go`. The build succeeds
and `go version -m` on the binary embeds `golang.org/x/net v0.55.0`,
`golang.org/x/crypto v0.52.0` and the `go.opentelemetry.io/otel` set at
v1.43.0. The source has no `vendor/` directory, so `go get` + `go mod
tidy` is sufficient. Advisory IDs and fixed boundaries verified against
the OSV database.
After this change the shipped binary carries no Go advisory with an
available fix; the only residual scanner entries are in
`k8s.io/kubernetes` with no upstream fix, which no version bump can
close.
Closes#2885
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the Kamaji image version to a newer patch release.
* Refreshed several bundled Go dependencies to newer pinned versions
before building.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The etcd suite still failed every run after the clean-after-each rework:
etcd-empty-backup (first to run) passed, but etcd-basic and etcd-backup-schedule
wedged at EtcdCluster WaitingForFirstQuorum until the chart's helm --wait timed
out (InstallFailed).
Root cause, from the failed run's crust-gather snapshot: the StatefulSet's
data-etcd-{0,1,2} PVCs are retained on uninstall and the drain never reclaimed
them, so each later scenario rebound the first scenario's volumes (identical PV
UIDs and a single creationTimestamp across all three scenarios) and booted the
previous etcd's data dir and member identity — never forming a fresh first
quorum. The pods were individually Ready (StatefulSet readyReplicas: 3) with all
certs valid, which is why it looked like an operator/network issue rather than
stale storage.
Delete the data PVCs (app.kubernetes.io/managed-by=etcd-operator) in
etcd_cleanup and wait for them to clear alongside the HR and DataStore, so each
scenario starts from empty storage.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The viewer (readonly) S3 credential can still upload: the seaweedfs-cosi-driver
does not yet honor accessPolicy=readonly (cozystack/cozystack#1921, blocked
upstream on kubernetes-sigs/container-object-storage-interface#244). The whole
cozystack chain is correct — Bucket.spec.users.viewer.readonly: true binds the
viewer BucketAccess to the tenant-root-readonly BucketAccessClass whose
parameters.accessPolicy is readonly — but the driver grants the identity write
access anyway (verified live on dev10). The BATS original asserted the same
denial; its retries and toothless gate hid the failure, which this migration
surfaces.
Keep exercising the path and verifying admin read/write + viewer read, but emit
a ::warning:: instead of failing when the viewer can upload, so the suite is not
blocked on an unimplemented upstream feature. TODO(#1921): restore the hard
failure once the driver enforces read-only access.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- mongodb/redis: assert status.readyReplicas instead of status.replicas, so the
StatefulSet check verifies pods are actually Ready (matches openbao/qdrant).
- kuberture: poll for the asynchronously-populated target annotation before
asserting the annotation shape (it was a one-shot test under set -e that could
race on a slow cluster), and guard the probe-Pod phase capture so the
describe/logs diagnostics still run when the Pod never materializes.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw port capped the harbor-test-registry BucketClaim bucketReady wait
at the default 5m, half the 600s the BATS original used. seaweedfs/COSI
provisioning can lag several minutes on a loaded runner, so the dependent
harbor-test-system HelmRelease keeps its 10m budget while the BucketClaim it
depends on was capped at 5m — a latent false-flake. Restore 10m
(docs/agents/e2e-testing.md section 6).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The openbao suite's finally deleted PVC data-openbao-test-0 while the
StatefulSet pod still mounted it; the kubernetes.io/pvc-protection finalizer
blocked removal and the delete hit its 5m timeout, failing the suite on every
run (confirmed in CI runs 28084122948 and 28025182013). Delete the OpenBAO CR
first so the HelmRelease/StatefulSet/pod tear down and release the PVC, then
remove the now-unmounted PVC — mirroring the BATS teardown order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run_kubernetes_test applies the tenant Kubernetes CR imperatively (kubectl), so
Chainsaw's auto-cleanup — which only tracks resources it apply/create-d — never
reclaims it. On success the script's tail deleted it, but on failure nothing did:
cozy_cleanup() used to be invoked by hack/cozytest.sh's trap, and the Chainsaw
harness has no equivalent caller, so a failed kubernetes-latest/previous run
leaked the tenant cluster (tens of GiB of worker PVCs) into the shared
tenant-test namespace and starved later suites.
Add a `finally` block to both kubernetes suites that runs cozy_cleanup. `finally`
always executes, after the `catch` phase, so the ordering matches the BATS
harness: pass -> cleanup; fail -> crust-gather (tenant trap + host catch) ->
cleanup. cozy_cleanup stays best-effort (|| true) so a slow tenant teardown never
flips a passing test red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Convert hack/e2e-apps/backup.bats.disabled to a Chainsaw test under
hack/e2e-chainsaw/backup/. It stays opt-in: the chainsaw-test.yaml.disabled
filename keeps it out of chainsaw's default discovery and out of
hack/select-e2e.sh enumeration, matching the previous disabled state.
The declarative readiness chain (Bucket -> VirtualMachine -> BackupJob) becomes
assert operations; the imperative Velero-backup + S3 verification stays a single
script step, mirroring the bucket suite. README documents the opt-in run command
and the backup-infra preconditions.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd Chainsaw scenarios share the singleton release name 'etcd' and
the cluster-scoped DataStore/<namespace> the etcd chart provisions for Kamaji.
On teardown Kamaji leaks its kamaji.clastix.io/TenantControlPlane finalizer on
that DataStore — once the backing etcd is gone the reconcile that would drop it
cannot complete — so the DataStore wedges in Terminating and the etcd
HelmRelease uninstall hangs. In CI this errored every etcd case: the first
scenario's stuck teardown blocked the next scenario's install (HelmRelease
never Ready, "DataStore status: Terminating", etcd pods refusing :2381 because
the CA issuer had no key material).
Make teardown explicit instead of relying on Chainsaw auto-cleanup (which was
waiting on the wedged uninstall): set skipDelete on each Test and have every
scenario's ensure-clean drain the previous one — delete the etcd and
force-clear the orphaned finalizer (no TenantControlPlane uses this
per-namespace test DataStore) in a loop until the HelmRelease and DataStore are
both gone. Each scenario then installs from a clean slate.
Verified on a dev cluster: all three scenarios pass back-to-back where they
previously cascaded.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Completes the SHA-pinning of pull-requests.yaml — the three
setup-buildx-action references were the only remaining tag-pinned (@v3)
actions after the merge. Pin them to v3.12.0's commit SHA so every action in
the workflow is digest-pinned.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the parallel-matrix PR build (#2983) with main, which since this branch
diverged landed a competing single-VM build restructure (#2939) and
test-impact-analysis E2E (#2559). Per maintainer decision, #2983's matrix build
supersedes #2939:
- .github/workflows/pull-requests.yaml: keep #2983's
plan/checks/build(matrix)/build-talos/finalize jobs; keep main's newer
resolve_assets/e2e (carrying #2559 test-impact E2E). The build->e2e artifact
interface (pr-patch + talos-image) is identical on both sides, so e2e depends
on finalize/build-talos with no input rewiring. main's top-level permissions
and SHA-pinned action refs applied across all jobs.
- All other files merged cleanly (only pull-requests.yaml conflicted).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two default golden-image URLs had drifted from their canonical source:
- alpine-3.21 was pinned to patch 3.21.6, but Alpine's CDN now ships
3.21.7 and prunes old patches over time, so the pin would eventually
404. Bump it to 3.21.7.
- opensuse-leap-16.0 pointed at the Open Build Service staging tree
(repositories/openSUSE:/Leap:/16.0:/Images/) rather than the canonical
released-product tree. Switch to distribution/leap/16.0/appliances/,
which serves the identical image from a more stable location.
The other 14 default image URLs use floating current/latest pointers on
canonical vendor hosts and need no change. Both new URLs were verified to
return HTTP 200 and serve valid QCOW2 images.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mariadb-operator hardcodes the mysqld-exporter liveness and readiness
probe as an httpGet on port 9104, which kubelet dials over the pod's IPv4
address. The exporter's default [::] (IPv6 wildcard) listener intermittently
refuses the IPv4 connect on single-stack-IPv4 data planes, so readiness fails
shortly after start, the pod flaps to NotReady, and the metrics Service keeps
the pod in notReadyAddresses — its endpoint never populates.
Pin the exporter to the IPv4 wildcard via --web.listen-address=0.0.0.0:9104 so
the kubelet probe connects deterministically. The operator appends user args to
its defaults and does not itself set --web.listen-address, so there is no
duplicate-flag conflict.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
make unit-tests already runs every hack/*.bats via bats-unit-tests
(hack/cozytest.sh, a pure-shell bats runner that needs no bats binary), so the
explicit step that apt-installed bats and re-ran build-matrix_test.bats /
select-e2e_test.bats was redundant. The selector suites still run in CI through
make unit-tests.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Fixes two failure modes in the COSI control plane built from upstream
container-object-storage-interface v0.2.2, both carried as image
patches.
**BucketClaim never converges to ready (#3032).** The controller does
not watch Bucket objects, and the generic controller drops no-op resync
deltas, so a BucketClaim whose backend Bucket flips to ready *after* the
first reconcile stayed `bucketReady=false` indefinitely — BucketAccess
was never granted and the consuming app (bucket, harbor) got no
credentials. The existing re-read of the live Bucket only ran during a
reconcile that nothing re-triggered. The controller now returns an error
while the Bucket is not ready, forcing a rate-limited requeue that
re-reads the Bucket until its readiness propagates, so the claim
converges deterministically instead of racing the async sidecar.
**Slow provisioner failover on pod recreate (part of #2429).** The
controller and the per-driver sidecar run single-replica, so leader
election adds no availability — only failover latency. The upstream 150s
lease made a recreated provisioner pod stall reconciliation for up to
~150s. The lease is lowered to the standard Kubernetes values
(15s/10s/2s), so a recreated pod resumes in ~15s. The stale unix-socket
CrashLoop in #2429 was already resolved by the seaweedfs-cosi-driver
v0.3.1 self-heal; the silent "Bucket ready with no backend bucket"
finding on that issue is a separate driver bug and is left tracked
there.
Coverage: a unit test in the patched controller pins the
requeue/propagation contract (fails without the requeue), and the
bucket/harbor e2e suites now assert `bucketReady=true` on a tight bound
instead of tolerating the lag, with scoped diagnostics on the failure
path.
Closes#3032
Refs #2429
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved object storage bucket readiness propagation so bucket claims
now track the backend bucket’s actual `bucketReady` status and requeue
when readiness isn’t yet observed.
* Reduced and made readiness waits more reliable in end-to-end checks,
with explicit match conditions and clearer debug output (including claim
YAML, backend bucket list, and controller pod status).
* Shortened leader-election timing to speed up recovery after pod
restarts.
* **Chores**
* Updated automated PR labeling to map the object storage controller
scope to the storage area label.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Make it unambiguous where each kind of contribution lands:
| Contribution | Destination |
|---|---|
| Bug | issue here (`kind/bug`) |
| Concrete, scoped feature | issue here (`kind/feature`) |
| Question / early idea | GitHub Discussions |
| Cross-cutting / architectural change | design proposal in
cozystack/community |
| Governance / process / community | cozystack/community |
## Changes
- Add `.github/ISSUE_TEMPLATE/config.yml` — a chooser
(`blank_issues_enabled: false`) linking to Discussions, the community
design-proposal process, community governance issues, and chat.
- Add `.github/ISSUE_TEMPLATE/feature_request.md` (`kind/feature`) with
an up-front gate (question → Discussion; cross-cutting → design
proposal) and a scope checklist.
- `CONTRIBUTING.md`: add a "Where do I file?" routing table.
- `README.md`: feature requests now point at issues instead of a
Discussions category, reconciling a contradiction with the actively-used
issue tracker.
## Notes
- `blank_issues_enabled: false` removes the blank-issue escape hatch to
keep triage clean — easy to flip if you'd rather keep it.
- Pairs with a companion PR in cozystack/community (routing table +
dead-link fix).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added “Where do I file?” guidance to help route bugs, scoped feature
requests, questions/early ideas, cross-cutting design proposals, and
community/process topics.
* Updated the README “Contributions” section with clearer links and
pointers to the right place for each request type.
* **New Features**
* Added a structured “Feature request” issue template to collect
consistent, scoped details.
* Disabled blank issue submissions and provided curated entry points for
common support and discussion paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
`packages/apps/kafka` rendered the Strimzi `Kafka` CR with
`entityOperator.topicOperator: {}` / `userOperator: {}` and no
`resources`. Tenant namespaces ship a `LimitRange` that defaults
containers to `memory: 128Mi`, so the topic-operator and user-operator
JVM containers (Kafka 3.9.1 + fabric8/vertx) are **OOMKilled (exit
137)** on startup. The `<cluster>-entity-operator` Deployment never
becomes Ready (CrashLoopBackOff), so:
- `KafkaTopic` / `KafkaUser` CRs never reconcile — topic creation
effectively fails;
- the cluster-operator logs `Exceeded timeout of 300000ms while waiting
for Deployment ...-entity-operator ... to be ready`;
- on teardown, `KafkaTopic`s get stuck permanently with the
`strimzi.io/topic-operator` finalizer (no operator left to clear them),
blocking namespace deletion.
The Kafka broker + ZooKeeper come up fine — only the entity-operator
dies. The `apps.cozystack.io/Kafka` schema exposes no field for
entity-operator resources, so there is no user-side workaround.
## Fix
Set explicit `resources` (requests `256Mi`/`100m`, limit `512Mi` memory)
on `entityOperator.topicOperator` and `userOperator` so they no longer
inherit the 128Mi namespace default. Adds a `helm-unittest` assertion
for the rendered limits.
## Testing
Reproduced on a clean cluster (Cozystack v1.4.3 and v1.5.0-rc.2; Strimzi
0.45.1, Kafka 3.9.1): both entity-operator containers `OOMKilled` at
128Mi → CrashLoopBackOff → `KafkaTopic` never `READY`. After bumping to
512Mi the entity-operator rolls cleanly (both containers Ready, 0
restarts) and the `KafkaTopic` reconciles to `READY=True`.
### Release note
```release-note
fix(kafka): give entity-operator (topic/user-operator) explicit resources so they no longer inherit the 128Mi tenant LimitRange default and OOMKill, which left KafkaTopics/KafkaUsers unreconciled.
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Kafka topic and user operators now use explicit CPU and memory
requests and limits, avoiding reliance on namespace-level defaults.
* **Tests**
* Added regression coverage to verify the Kafka entity operator’s topic
and user operator memory limits are explicitly set (512Mi).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Removes the `Release E2E` workflow
(`.github/workflows/release-e2e.yaml`), which ran the full bats suite on
every release tag.
**Why it's redundant.** `release-e2e.yaml` was a non-blocking alarm —
its own header states *"a failure here is an alarm, not a release
blocker."* The coverage it provided is already delivered by the
release-cut PR: the "Prepare release" commit bakes image digests into
`packages/core/` (installer, platform, testing) and other packages, and
`select-e2e.sh`'s full-suite pattern escalates any `packages/core/`
change to the **full** suite. So the release PR already runs the whole
suite against the commit the tag will point at.
**Why now.** The workflow has never been green since it was added
(2026-05-02) — every tag run (`v1.5.0-rc.2`, `-rc.3`, `v1.5.0`) failed
at the build step. Even after #2973 fixed the registry login earlier
today, the next run still failed at *Build Talos image*. A
perpetually-red, non-gating duplicate is pure noise.
**Changes**
- Delete `.github/workflows/release-e2e.yaml`.
- Drop `release-e2e` from the full-suite pattern in
`hack/select-e2e.sh`.
- Repoint the workflow-trigger unit test in `hack/select-e2e_test.bats`
to `pull-requests.yaml` (the remaining E2E workflow), preserving
"workflow edit → full suite" coverage. All 11 `select-e2e_test.bats`
cases pass.
- Update `docs/agents/e2e-testing.md` to describe the release-cut PR as
where the full suite runs.
Refs #2397.
### Release note
```release-note
ci(tests): remove the redundant Release E2E workflow; the full E2E suite already runs on the release-cut PR
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Removed automated release end-to-end testing workflow
* Updated E2E test selection logic to reflect testing infrastructure
changes
* **Documentation**
* Updated E2E testing procedures documentation to reflect current
testing approach
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Run hack/build-matrix_test.bats and hack/select-e2e_test.bats in the checks
job so a selector parse regression is caught in CI rather than shipping.
- curl -fsSL the flux install so an HTTP error page is not piped to sudo bash.
- git apply --3way in finalize so two fragments touching adjacent lines of a
shared file merge instead of the second apply aborting the whole step.
- Add least-privilege permissions: {contents: read} to the plan and checks jobs.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The matrix selector's full_rebuild_pattern omitted the root Go module dirs
(api/, cmd/, internal/, pkg/). Eight matrix image Dockerfiles COPY those dirs,
so a PR changing only root Go source emitted an empty matrix, skipped the build
job, and let e2e install stale digests — the Go change was never exercised. Add
the four dirs to the pattern and pin it with bats cases for api/- and pkg/-only
diffs forcing the full matrix.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The kafka e2e test pinned both the kafka and zookeeper components to the
legacy "nano" preset (250m CPU / 128Mi memory). 128Mi cannot hold the
Strimzi Kafka 3.9.1 and ZooKeeper JVMs: the brokers OOMKill on startup
(exit 137) and CrashLoopBackOff, so the pods never reach Ready, the Kafka
CR and its HelmRelease never go Ready, and the 5m readiness wait times
out. It surfaced as a flake because 128Mi sits right at the JVM-start
threshold — occasionally a broker passes readiness before its RSS crosses
the cgroup limit.
Raise both components to "c1.small" (1 CPU / 1Gi), the chart default,
which comfortably fits the JVMs. The e2e sandbox VMs are 8 CPU / 24Gi, so
1Gi across the four pods is well within budget.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).
Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.
RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
ovn-controller runs inside the ovs-ovn DaemonSet's openvswitch container,
sharing it with ovsdb-server and ovs-vswitchd. It is CPU-bound while
programming OpenFlow for newly created logical ports. Under a burst install
(many pods created at once) it competes for CPU in that single container and
CFS-throttles against the 2-core limit, lengthening flow-programming lag that
leaves freshly created pods unreachable from their node until the flows land.
Raise the ovs-ovn CPU limit from 2 to 4 cores to give ovn-controller headroom
to program flows promptly under burst, and lift the CPU request from 10m to
100m so it keeps a guaranteed share under node CPU contention. Memory is left
unchanged. Add a helm-unittest guard pinning the new CPU limit and request.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Extend the host->pod data-plane capture with the three datums that make an
OVN incremental-processing lag dispositive on the next recurrence, all
diagnostic-only and gated on the existing on-failure path:
- ovn-controller.log from the ovs-ovn pod (grep for physical_flow_output,
if_status_mgr, took <N>ms, recompute, long poll interval; plus a bounded
tail) -- shows whether ovn-installed was set before the LSP local-delivery
flow was programmed.
- per-interface ovn-installed flag + ovn-installed-ts on the pod's OVS
interface, looked up by iface-id <pod>.<ns>.
- ovs-ovn cgroup cpu.stat (nr_throttled / throttled_usec, cgroup v2 with v1
fallback) -- shows whether ovn-controller was CPU-starved.
Node-global captures land in node-<node>.txt, the per-interface flag in the
per-pod file. Every command stays timeboxed and || true, no behavior or
exit-code change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.
On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):
- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
enable-host-legacy-routing the host->local-pod path traverses the kernel
netfilter stack, not cilium BPF, so this is the authoritative conntrack
table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
bounded monitor --type drop, and hubble dropped verdicts when present
(complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.
Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fluent-bit DaemonSet crash-loops on a loaded install, timing out its
HelmRelease and cascading into velero and backupstrategy-controller
DependencyNotReady failures. The subchart default liveness/readiness probes
(initialDelaySeconds 0, timeoutSeconds 1, failureThreshold 3) target the HTTP
server on :2020, but that server does not bind until AFTER the kubernetes
filter's 'testing connectivity with API server' step. Under a concurrent
install the kube-API can be unreachable for ~10s, delaying the :2020 bind,
while the single-threaded HTTP server can miss a 1s timeout when the engine is
busy. The probes then fire into a connection-refused port and liveness-kill
fluent-bit into a restart loop.
Give both probes load headroom (initialDelaySeconds 15, timeoutSeconds 5,
failureThreshold 6) so the API-gated late bind and busy-engine blips are
tolerated. The httpGet target is unchanged (path / on port http = :2020). Add
a helm-unittest pinning the relaxed timing and the preserved target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.
The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.
Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
While the backend Bucket is not ready the controller returns an error to
force a requeue, but that error is only logged at klog.V(3) and is
suppressed at default verbosity, so an operator cannot see why the claim
is stuck. Record a WaitingForBucket Warning event on the BucketClaim on
each requeue (event aggregation folds the repetition) and still return
the error so the requeue happens as before. Pin the requeue contract in
the test by asserting the error carries the stable "not ready yet"
substring, not merely that it is non-nil.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Run `go mod tidy` to move github.com/fluxcd/pkg/apis/meta from
indirect to direct require: it became a direct import after the
SecretRef type fix.
- Refine the --platform-source-secret flag docstring and the matching
values.yaml comment: drop identity.pub from the SSH required-keys
list (Flux source-controller only consumes identity + known_hosts)
and mention bearerToken as an alternative to username+password for
Git over HTTPS.
- Note in values.yaml that the Secret must exist in cozy-system before
the operator reconciles; otherwise source-controller reports a
failed condition until the Secret is created.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Aleksei Sviridkin