mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-10 14:34:58 +09:00
bfaac1ef126da43aeaac90ee9e75353d405be94e
4418
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov DaniilandClaude Opus 4.8
|
bfaac1ef12 |
test(e2e): enrich host crust-gather snapshot with helm + collect logs
cozytest.sh's parent-shell EXIT trap snapshots the HOST cluster on a failed test (nested tenant clusters are captured separately, by run-kubernetes.sh's in-subshell trap, while their port-forward is still alive). That host snapshot had three gaps: the collect's output went to /dev/null so a failed collect left no clue why; Helm release storage (kind Secret) was dropped along with every other Secret, losing the revision history that explains a HelmRelease stuck mid-install; and the node host-log scanner burned the collect's --duration deadline on debug pods PodSecurity can never admit, forcing a non-zero exit even though the API tree was already on disk. - factor the collect into a _cozy_collect helper that captures stdout/stderr/rc to a sibling .collect.log and prints a per-target status, so a failed collect leaves evidence in the artifact instead of vanishing; bound it with `timeout -k 30 5m` so a degraded cluster can't stall the job - classify a non-zero collect that still wrote a resource tree as PARTIAL rather than FAILED, so a usable snapshot isn't discarded over the label (capture $? as the first statement in the else — after `fi` it would be the if-compound's 0, not the collect's exit code) - add a dedicated include-only collect (_cozy_collect_helm) for Helm release secrets (sh.helm.release.*), so the artifact carries each release's revision history and status - pass --disable-additional-logs so the host-log scanner doesn't burn the deadline on debug pods that can never be admitted; container/pod logs come from a separate path and are unaffected, and Talos host logs aren't reachable that way anyway crust-gather puts all CLI filters in one FilterList evaluated with .all(), so `--include-kind Secret --include-name <re>` keeps a Secret only when both match — verified from source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
4404c5c5eb |
fix(etcd): pin tenant etcd to v3.5.31 to restore consistent watches (#3122)
## What this PR does Tenant Kubernetes control planes get their etcd from the `extra/etcd` chart, which creates an `EtcdCluster` (`etcd.aenix.io/v1alpha1`) reconciled by etcd-operator v0.4.5. That operator hardcodes the etcd image to `quay.io/coreos/etcd:v3.5.12`, and the v1alpha1 CRD exposes no version field, so every tenant etcd runs 3.5.12. kube-apiserver only enables the `RequestWatchProgress` storage feature on etcd >= 3.5.13 (or >= 3.4.31). Kubernetes >= 1.31 enables `ConsistentListFromCache` — locked-to-true in 1.35 — which depends on it. On etcd 3.5.12 every consistent watch therefore fails inside the stream with HTTP 500 (`the required storage feature RequestWatchProgress is disabled`), which breaks watch-based clients of aggregated APIs in modern (k8s 1.35) tenants. This pins the etcd container image to a >= 3.5.13 patch release (v3.5.31) by setting `image` on the `etcd` container in the EtcdCluster's `spec.podTemplate.spec`. etcd-operator v0.4.5 strategically merges `podTemplate.spec` over its generated pod spec (matched by container name), so the override takes effect with no operator change — the same mechanism the chart already uses for the metrics port, probes, and resources. It restores consistent watches without waiting on the etcd-operator v1alpha2 migration (#2859), which is the longer-term fix. A helm-unittest suite asserts the etcd container carries the pinned image and is no longer left on the operator default v3.5.12. Addresses #3080. ### Upgrade impact On merge, etcd-operator reconciles every existing tenant `EtcdCluster` and rolls its StatefulSet from 3.5.12 to 3.5.31. This is a safe in-minor rolling patch upgrade (no etcd storage-format change, quorum preserved by the operator's rolling strategy), so expect a brief, sequential etcd pod restart per tenant and no data migration. ### Release note ```release-note fix(etcd): pin tenant etcd to v3.5.31 so kube-apiserver can enable RequestWatchProgress (consistent watches) on Kubernetes 1.35 tenants ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Pinned the bundled etcd image to `quay.io/coreos/etcd:v3.5.31` to improve compatibility with Kubernetes watch-related storage behavior. * Prevented deployments from reverting to the older default etcd image. * **Tests** * Added a new test suite to verify the template uses the expected pinned etcd image and blocks regressions. * **Chores** * Updated dependency automation to recognize and manage the pinned etcd image version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
4db565282b |
fix(postgres): garbage-collect init-job after completion (#3055) (#3069)
## Problem After deleting a `Postgres` app CR, the chart-managed init Job `<release>-init-job` and its completed Pod remain in the tenant namespace indefinitely (reported in #3055). The init Job is a Helm hook (`post-install,post-upgrade`) whose only delete policy is `before-hook-creation`. That policy deletes the previous hook only right before a new one is created (next upgrade) — it never fires on `helm uninstall`. With no `hook-succeeded` and no TTL, the completed Job and Pod leak. ## Root cause `packages/apps/postgres/templates/init-job.yaml` — the Job has no garbage-collection mechanism for its completed state. ## Fix Add `ttlSecondsAfterFinished: 120` to the Job spec. The native Kubernetes TTL controller then GCs the Job and its Pod ~2 minutes after completion, regardless of the Helm hook lifecycle. The init Job is a one-shot, idempotent reconciler of users/databases/roles (`init-script.yaml`) — once it has run it is not needed until the next upgrade (when Helm recreates it), so GC-ing the finished Job is safe. This approach deliberately **does not** touch the hook-delete-policy / hook-timeout semantics, which the `bootstrap.recovery` path depends on (see the header comment in `init-job.yaml`). Under `bootstrap.enabled` the Job is skipped entirely, so the change is inert there. ## Tests Added a `helm-unittest` suite `packages/apps/postgres/tests/init_job_cleanup_test.yaml`: - asserts the init Job renders `spec.ttlSecondsAfterFinished: 120` - asserts the init Job is skipped under `bootstrap.enabled: true` Verified locally with `helm template` (TTL present on default render; Job absent under bootstrap recovery). ## Test plan - [x] `helm template` renders `ttlSecondsAfterFinished: 120` on the init Job - [ ] `helm unittest packages/apps/postgres` green in CI - [ ] e2e `test-apps-postgres` green Fixes #3055 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Postgres bootstrap init Jobs now automatically clean up after completion using a post-finish TTL. * **Tests** * Added a new integration test suite to verify the init Job renders with the expected cleanup TTL, and that the init Job is skipped when bootstrap is enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
49555e562b |
fix(openbao): garbage-collect the data PVC on delete (#3058) (#3073)
## What this PR does
After deleting an `OpenBAO` app CR, the StatefulSet data PVC
(`data-<release>-0`)
was left orphaned in the tenant namespace, consuming storage
permanently. The
OpenBAO app renders a Flux `HelmRelease` over the vendored upstream
OpenBAO chart,
and StatefulSet-templated PVCs are never garbage-collected on
StatefulSet deletion
by default — the chart set no `persistentVolumeClaimRetentionPolicy`.
This passes a retention policy through the HelmRelease values. The
upstream chart
renders it onto the StatefulSet from
`server.persistentVolumeClaimRetentionPolicy`
(verified in the vendored `server-statefulset.yaml`) — **not** from
`server.dataStorage.*`:
```yaml
server:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Delete
whenScaled: Retain
```
A `test` target is added to the package Makefile so the new
helm-unittest suite
(asserting the passthrough) actually runs in CI —
`hack/helm-unit-tests.sh` only
runs packages that define a `test` target.
> [!WARNING]
> **Deleting an OpenBAO app now permanently removes its data PVC and the
data it holds.**
> This is intentional and fixes the storage leak. `whenScaled: Retain`
is kept on
> purpose so scaling an HA/raft cluster down does **not** destroy a
member's data —
> only full deletion of the release reclaims storage. Back up before
deleting.
Fixes #3058.
### Release note
```release-note
fix(openbao): set persistentVolumeClaimRetentionPolicy so the data PVC is removed when an OpenBAO app is deleted, instead of being orphaned. WARNING: deleting an OpenBAO app now permanently removes its data PVC and the data it holds (scale-down still retains data).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable PVC retention behavior for OpenBao: storage is
deleted when the release is removed and retained when scaling down.
* **Tests**
* Added a Helm unit testing workflow to run chart/unit tests.
* Introduced assertions to confirm PVC retention settings are correctly
passed through and rendered as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
scooby87
|
959a9b1fdb |
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062) (#3078)
## What this PR does After deleting a `Kubernetes` app CR / tenant, the secret `<release>-datastore-config` kept the finalizer `finalizer.kamaji.clastix.io/datastore-secret`, which was never removed — **blocking deletion of the entire namespace** (observed stuck in `Terminating` for 115 minutes). Kamaji adds this finalizer to the datastore connection Secret and normally strips it as part of TenantControlPlane (TCP) deletion; if the TCP is force-deleted or Kamaji does not reconcile in time, the finalizer hangs and the Secret can never be garbage-collected, wedging the namespace. This extends the existing `pre-delete` cleanup hook (`packages/apps/kubernetes/templates/delete.yaml`): - **Step 4** now deletes the TenantControlPlane with `--wait=true --timeout=120s` so Kamaji's own finalizer-cleanup chain has a chance to run before we fall through. - **Step 4b (backstop)** patches `metadata.finalizers: null` on `<release>-datastore-config` only if it still exists. By this point the TCP is gone, the Secret is owner-ref'd to the deleted TCP and only awaiting GC, so clearing the finalizer merely lets GC reap it — it cannot strip a finalizer while a live TCP still needs the datastore. The Secret holds only connection config, not tenant data (which lives in the external etcd datastore), so no data is lost. - **RBAC** adds a `secrets` rule scoped via `resourceNames` to exactly `<release>-datastore-config`, verbs `get`/`patch` only. The Job is hardened: non-root (uid 65534), read-only root fs, all caps dropped, `seccompProfile: RuntimeDefault`, writable `/tmp` for kubectl cache. Failures at both steps are logged (the Step 4b warning explicitly mentions the namespace may stay Terminating) but the hook still completes so it cannot itself block teardown. This is the most impactful of the create→delete leak sweep findings: without it, tenant teardown hangs and the create→delete cycle requires manual intervention. Fixes #3062. ### Release note ```release-note fix(kubernetes): clear the leftover Kamaji `finalizer.kamaji.clastix.io/datastore-secret` finalizer on the datastore-config Secret during tenant teardown, so deleting a Kubernetes app / tenant no longer leaves the namespace stuck in Terminating. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Extended the pre-delete cleanup workflow to more reliably remove `TenantControlPlane` and handle slow deletions with a bounded wait. * **Security** * Hardened the cleanup job by running as non-root with restricted security settings, dropped capabilities, and a safe writable temp location. * **Bug Fixes** * Added a conditional backstop to remove leftover `finalizers` from the relevant datastore Secret so namespace termination isn’t blocked. * **Tests** * Added a YAML test suite to validate the wait-and-patch behavior and ensure the patch runs only after the `TenantControlPlane` is gone. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
080ceac9c0 |
fix(gateway): clean up orphaned ACME account secret on tenant module delete (#3089) (#3093)
## What this PR does When the `gateway` tenant module is disabled (its HelmRelease uninstalled), cert-manager's ACME account private-key secret `cozystack-acme-account` was left orphaned in the tenant namespace (label `app.kubernetes.io/managed-by: cert-manager`, no `ownerReferences`). This adds a `post-delete` cleanup hook (`packages/extra/gateway/templates/hooks/cleanup-acme.yaml`) that deletes the secret by name. Verified on a live cluster that `cozystack-acme-account` is referenced by exactly one Issuer — the gateway module's own `cozystack-gateway` Issuer — so it is gateway-specific and safe to remove on gateway teardown (it is not shared with other modules, which use the cluster-wide `letsencrypt-prod` issuer). The cleanup Job is hardened: non-root (uid 65534), read-only root fs, all capabilities dropped, `seccompProfile: RuntimeDefault`, resource requests/limits; RBAC scoped via `resourceNames` to exactly `cozystack-acme-account` (get/delete). A failure is logged but never blocks teardown. No user data is affected — only cert-manager's ACME account key (re-registered on next enable). Fixes #3089. ### Release note ```release-note fix(gateway): clean up the orphaned cert-manager ACME account secret (cozystack-acme-account) left behind when a tenant's gateway module is disabled. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added automatic cleanup after gateway removal to delete the stored ACME account secret in the release namespace. * Included a dedicated, restricted cleanup job with the required permissions to run safely during uninstall. * **Bug Fixes** * Made the gateway test target always runnable. * Cleanup now ignores missing secrets and warns on failure instead of stopping uninstall flows. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
90aa850c95 |
ci(labeler): map cozystack-ui and console scopes to area/dashboard (#3117)
## What this PR does The cozystack-ui SPA was vendored into the monorepo in #2963 and now lives under `packages/system/dashboard`. PRs touching the UI are likely to use `cozystack-ui` or `console` as their Conventional Commits scope, neither of which the PR auto-labeler recognizes today — so they fall through to `area/uncategorized`. This maps both scopes to the existing `area/dashboard` (already described as "dashboard / UI"). No new label is introduced; `.github/labels.yml` is unchanged. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Expanded automatic PR labeling to add the `area/dashboard` label when a PR title scope is `cozystack-ui` or `console`. * Existing label rules and parsing behavior remain unchanged for all other scopes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6896a2a4db |
feat(kubernetes): expose tenant apiserver authentication in app values (#3123)
## What this PR does The managed `Kubernetes` app (Kamaji-backed tenant control planes) exposed no way to configure the tenant `kube-apiserver` authentication through the application values — there was no knob for OIDC, the structured `AuthenticationConfiguration`, extra apiserver flags, or extra volumes. The only workaround was patching the underlying `KamajiControlPlane` out-of-band, which is invisible to GitOps and lost when the cluster is recreated from values. This adds three opt-in fields under `controlPlane.apiServer`, all empty by default so existing clusters render unchanged: - `extraArgs` → `KamajiControlPlane.spec.apiServer.extraArgs` - `extraVolumeMounts` → `spec.apiServer.extraVolumeMounts` - `extraVolumes` → appended to `spec.deployment.extraVolumes` This unblocks OIDC SSO via the `--oidc-*` flags and, more importantly, structured authentication: `--authentication-config` mounted from a ConfigMap/Secret plus `--requestheader-uid-headers`, which is the only way to map a user UID for aggregated/extension apiservers. Because the control-plane Deployment runs on the management cluster, the passthrough is validated to keep tenant and management isolated: `extraVolumes` accept only `configMap`/`secret` sources (host-reaching sources such as `hostPath`/`csi` and token projection via `projected` are rejected); each volume needs a unique, non-empty name and exactly one source; the reserved `talos-ca`/`talos-tls-cert` names are rejected; and each `extraVolumeMount` must reference a user-declared `extraVolume`, so the chart-managed talos secret volumes cannot be mounted. Secret/ConfigMap references resolve in the tenant's own management-side namespace and are intentionally allowed — this stays within the tenant's existing security domain. Generated artifacts (`values.schema.json`, README, Go types + deepcopy, ApplicationDefinition) are regenerated from the values annotations. Helm unit tests cover the default no-op render, each field's passthrough, and every rejection path. Addresses #3074. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(kubernetes): expose tenant apiserver authentication via controlPlane.apiServer.extraArgs / extraVolumes / extraVolumeMounts, enabling OIDC and structured AuthenticationConfiguration on tenant clusters ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for customizing the Kubernetes API server with extra command-line arguments, additional volume mounts, and extra volumes. * Updated chart values and schema so these options can be configured directly. * **Bug Fixes** * Added validation to prevent invalid or conflicting volume names and disallowed volume sources. * Ensured mounts only reference declared extra volumes and reserved control-plane names can’t be overridden. * **Tests** * Added coverage for default behavior, valid customization paths, and invalid configuration scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
80e3fa9cbe |
fix(dashboard): render error and unknown-type states instead of an infinite spinner (#3098)
## What this PR does Two console pages ordered their loading guard ahead of the terminal state, making the latter unreachable. Both are pre-existing upstream behaviors carried in by the cozystack-ui vendoring (#2963) and flagged as non-blocking findings in that PR's review. - **`ApplicationDetailPage`** checked `isLoading || !instance || !ad` before `if (error)`. On a failed GET (403/404/5xx) `isLoading` is false and `instance` stays undefined, so the page rendered the loading spinner forever and never reached the "not found" branch. The error branch now runs first. - **`ApplicationListPage`**'s `defsLoading || (!ad && plural)` guard stayed true for any unrecognized `plural` once definitions had loaded, so the "Unknown application type." branch was unreachable. The guard now only spins while definitions are loading. ## Verification - Added co-located regression tests for the failed-GET (detail) and unknown-plural (list) paths. - `pnpm typecheck` clean; full console suite green (`pnpm test`: 317 passed, +4 new). ```release-note fix(dashboard): the console now shows a not-found message on a failed application GET and an unknown-type message for an unrecognized resource, instead of spinning forever ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved loading and error handling on application pages so users see the correct message at the right time. * “Unknown application type” now appears only after application definitions finish loading, instead of being masked by a loading state. * Application detail pages now show a “not found” message when the requested app can’t be loaded, rather than continuing to display a spinner. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
df221f98d4 |
ci(labeler): map cozystack-ui and console scopes to area/dashboard
The cozystack-ui SPA was vendored into the monorepo (#2963) and now lives under packages/system/dashboard. Map its likely PR title scopes to the existing area/dashboard so UI PRs no longer fall through to area/uncategorized. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
61a574393a |
build(dashboard): pin console image builder to pnpm 11.1.2 and repoint vendor doc link (#3099)
## What this PR does Two post-vendor cleanups from the #2963 review: - **Pin the console image builder to pnpm 11.1.2.** The `Containerfile` ran `corepack prepare pnpm@latest`, so the container build could drift to a newer pnpm major than the `pnpm@11.1.2` pinned in `package.json` and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the builder makes the image build reproducible and consistent with CI. - **Repoint a stale doc link.** `docs/storage-immutability.md` referenced the now-vendored `cozystack/cozystack-ui#6`; it now points at the in-tree console path. ## Verification - No source change; `pnpm typecheck` + `pnpm test` unaffected. - Builder pnpm version now matches `package.json`'s `packageManager` field. ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified the storage immutability documentation by updating the UI reference to an in-repo path. * **Chores** * Standardized the build environment to use a specific pnpm version, helping keep installs and builds consistent. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
aec16212ee |
refactor(build): parallel matrix image builds on ephemeral runners (#2983)
## What this PR does Completes #2937. Replaces the single serial `Build` job — one ephemeral VM running `make build` (~26 images one after another) that wedged on the shared embedded buildkit under concurrent multi-PR load — with a parallel, diff-scoped matrix. **New job graph (regular PR path):** ``` plan changed files -> hack/build-matrix.sh -> scoped matrix + docs gate checks make unit-tests + test-controllers (gates finalize -> e2e) build one package per job, own docker-container buildx, uploads a per-package digest patch fragment [parallel] build-talos talos image + nocloud disk (its own leg; e2e always needs the disk) finalize download+apply all fragments, then build the installer LAST (its flux-push bundles the whole digest-patched tree), emit the unified pr.patch e2e consumes talos-image + pr.patch unchanged ``` **Why finalize is special:** `packages/core/installer`'s `image` target runs `flux push artifact --path=packages`, bundling the *entire* packages tree into the OCI artifact the operator pulls — so it must run after every other package's digest edit is applied, never as a peer matrix job. `talos` is likewise pulled out of the matrix (its image and nocloud disk share heavy `_out/assets` and build once). **Gating:** `finalize` (and therefore `e2e`, which needs it) depends on `checks`, so a red unit/controller test skips the expensive install+e2e path instead of running it for ~1h — the gating the old in-build-job tests gave, without re-coupling the cheap parallel builds. **Change-scoping:** `hack/build-matrix.sh` parses the root Makefile `build:` target (single source of truth) and emits only the packages a PR touches — most PRs build ~0–1 images — escalating to the full set on a shared-dependency change (`cozy-lib`, the build macros, `go.mod`, the build workflows). It fans `seaweedfs` out to `objectstorage-controller` (cross-package sidecar-tag dependency). 16 bats cases cover the selector. **SBOM:** `SBOM=1` wires buildx `--sbom` through `common-envs.mk` (off by default). It stays off in the matrix until an OCIR referrers/attestation probe confirms the registry accepts attestations. **Decommission:** the cache warmer (`build-main.yaml`) moves to the ephemeral pool, so no build path depends on the dedicated self-hosted runner. The runner remains referenced only by the e2e `debug` breakpoint path; fully de-registering it is a separate infra step. ### Validation - Local: `hack/build-matrix_test.bats` 16/16; `actionlint` clean (custom runner labels registered in `.github/actionlint.yaml`); `make -n` confirms `--sbom` is off by default and on with `SBOM=1`. - CI: this PR edits `pull-requests.yaml`, so `build-matrix.sh` forces a full matrix — the PR exercises the new pipeline on itself. First run was green through `plan → 25 parallel builds → build-talos → finalize → install → 21/22 app tests`; the only red was the pre-existing `bucket`/BucketAccess flake (install and the other 21 suites passed on the matrix-built tree), which validates the decomposition end-to-end. ### Follow-ups (tracked on #2937) - Probe OCIR referrers/attestation support; enable `SBOM=1` in the matrix if supported. - Infra: de-register the self-hosted build runner (the e2e `debug` breakpoint path is its only remaining user). ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Refined CI to a staged, artifact-based pipeline with gated checks, per-package parallel builds, and centralized patch assembly before building final images * Improved ephemeral runner performance and reliability (runner selection, longer timeout, build toolchain setup) * Added build-matrix auto-selection with targeted fan-out rules and “full rebuild” triggers * Updated CI lint configuration to recognize ephemeral runner labels * Introduced an optional SBOM toggle for build artifacts * **Documentation** * Updated e2e testing docs to better explain Test-Impact Analysis scope and gating semantics * **Tests** * Added Bats tests validating build-matrix selection and edge cases <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7a8d1267f2 |
fix(seaweedfs): point S3 consumers at the reachable S3 service (#3119)
## What this PR does The S3 Service is named with the release-independent `seaweedfs.name` helper (always `seaweedfs-s3`), but several in-chart consumers still resolved it through `componentName` (`<fullname>-s3`). Since the shipped release is `seaweedfs-system`, those consumers pointed at `seaweedfs-system-s3` — a Service that does not exist — so ingress-nginx returns 503 on the external S3 endpoint and the COSI provisioner cannot reach S3 in-cluster. This aligns every in-chart S3 consumer with the actual Service: - S3 ingress backend and iceberg ingress backend → `seaweedfs-s3` (fixes the 503 on the default `s3.ingress.enabled: true` path). - COSI provisioner in-cluster `ENDPOINT` (both the s3-gateway and filer-S3 branches) → `seaweedfs-s3` on the configured S3 port. These were also portless (defaulting to 443 against an S3 that listens on 8333), and the filer branch additionally targeted the filer Service, which serves no S3. The all-in-one ingress branch is intentionally left on `componentName "all-in-one"` (a different Service). Changes are captured as a downstream patch (`s3-service-name-consumers.patch`) wired into `make update`, and pinned by helm-unittest rendered under the production release name so the mismatch cannot recur. ### Release note ```release-note fix(seaweedfs): point the S3 ingress, iceberg ingress, and COSI provisioner endpoint at the actual seaweedfs-s3 service, fixing 503 on the external S3 endpoint ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated S3-related routing and in-cluster endpoints to use the correct service name and explicit port values. * Improved ingress behavior for newer Kubernetes versions while keeping compatibility with older versions. * Preserved the existing all-in-one routing path for S3 deployments. * **Tests** * Added coverage for S3 service naming, ingress backend routing, and COSI endpoint resolution to prevent regressions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
243ce65071 |
feat(kubernetes): expose tenant apiserver authentication in app values
Tenant Kubernetes clusters had no way to configure the kube-apiserver authentication through the app values: no knob for OIDC, the structured AuthenticationConfiguration, extra apiserver flags, or extra volumes. The only workaround was patching the underlying KamajiControlPlane out-of-band, which is invisible to GitOps and lost on recreation. Add three opt-in fields under controlPlane.apiServer, all empty by default so existing clusters render unchanged: - extraArgs -> KamajiControlPlane spec.apiServer.extraArgs - extraVolumeMounts -> spec.apiServer.extraVolumeMounts - extraVolumes -> appended to spec.deployment.extraVolumes This unblocks OIDC SSO via the --oidc-* flags and, more importantly, structured authentication: --authentication-config mounted from a ConfigMap/Secret plus --requestheader-uid-headers, which is the only way to map a user UID for aggregated/extension apiservers. Because the control-plane Deployment runs on the management cluster, the passthrough is validated to preserve tenant<->management isolation: - extraVolumes allow only configMap and secret sources; host-reaching sources (hostPath, csi, nfs, persistentVolumeClaim) and token projection (projected) are rejected, so a tenant with self-service access to the Kubernetes CR cannot mount management-node paths or project arbitrary tokens into the apiserver pod. - each extraVolume must have a unique, non-empty name and exactly one source; the reserved names talos-ca and talos-tls-cert are rejected to avoid a duplicate volume name in the rendered podspec. - each extraVolumeMount must reference a user-declared extraVolume; the chart-managed talos secret volumes cannot be mounted. Regenerate values.schema.json, README, the Go types and deepcopy, and the ApplicationDefinition. Add helm unittest coverage for the default (no-op) render, each field's passthrough, the name/source guards, and the mount-reference guard (valid and rejected inputs). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d1a2c1bc25 |
ci(renovate): track the pinned etcd image in extra/etcd
The etcd image in the extra/etcd EtcdCluster podTemplate is a hardcoded string that overrides etcd-operator v0.4.5's baked-in default. None of the enabled managers (gomod, dockerfile, github-actions) can see a literal image string in a Helm template, so the pin would silently age - and etcd 3.5.x releases are overwhelmingly security fixes. Add a custom.regex manager that matches the image line and resolves updates from the docker datasource, so future etcd patch releases surface as Renovate PRs like any other tracked dependency. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
26b3110c59 |
fix(etcd): pin etcd image to v3.5.31 for consistent watches
etcd-operator v0.4.5 hardcodes quay.io/coreos/etcd:v3.5.12 and the
v1alpha1 EtcdCluster CRD exposes no version field, so the tenant
control-plane etcd runs 3.5.12. kube-apiserver only enables the
RequestWatchProgress storage feature on etcd >= 3.5.13 / >= 3.4.31, and
Kubernetes >= 1.31 enables ConsistentListFromCache (locked-to-true in
1.35), which depends on it. On etcd 3.5.12 every consistent watch fails
inside the stream with HTTP 500 ("the required storage feature
RequestWatchProgress is disabled"), breaking watch-based clients of
aggregated APIs in modern tenants.
Override the etcd container image through spec.podTemplate.spec, which
the operator strategically merges over its generated pod spec (matched
by container name), pinning a >= 3.5.13 patch release. This restores
consistent watches without waiting on the etcd-operator v1alpha2
migration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
Aleksei Sviridkin
|
8a14de9abf |
fix(seaweedfs): point S3 consumers at the reachable S3 service
The s3-service-name.patch renamed the S3 Service to <name>-s3 (seaweedfs-s3), but several consumers still resolved it through componentName -> <fullname>-s3. name and fullname only coincide when the release name equals the chart name; the shipped release is seaweedfs-system, so they pointed at seaweedfs-system-s3 while the real service stayed seaweedfs-s3: - the S3 ingress backend and the iceberg ingress backend, which left ingress-nginx returning 503 on the external S3 endpoint. The COSI provisioner in-cluster ENDPOINT was wrong on the same axis and also on the port. Both the s3-gateway branch and the filer-S3 branch rendered a portless URL that defaults to 443, while the s3 service only listens on the S3 port (8333); the filer branch additionally targeted the filer service, which serves no S3. Both in-cluster branches now resolve seaweedfs-s3 on the configured S3 port. The default path (s3.ingress.enabled=true) is repaired by the ingress fix; the COSI in-cluster paths are latent until an operator disables the ingress. Resolve every consumer through the same name helper the Service uses, and pin the contract with unittests rendered under the production release name so the mismatch cannot recur. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
f5c408d2fc |
ci(renovate): bump indirect Go deps and enable OSV vuln alerts (#3026)
## What Makes Renovate open the dependency PRs that close the OpenSSF Scorecard `Vulnerabilities` advisories, instead of doing them by hand (as in #3019). - Enable **indirect** gomod updates (grouped into one rolling "indirect go modules" PR). Renovate skips indirect Go deps by default, which is why `golang.org/x/crypto`, `x/net`, and `x/sys` — the transitive deps carrying the 20 OSV advisories — never got a PR. - Enable **`osvVulnerabilityAlerts`** so vulnerable deps get dedicated, mergeable security PRs (labeled `security`, not the global `do-not-merge/hold`). ## Why #3019 wasn't enough #3019 bumped `x/crypto` in the **root** module only. The repo has 5 `go.mod` modules; the same advisories live in `token-proxy` (x/crypto v0.50.0), `kubevirt-csi-driver`, and `kubeovn-webhook` (x/net / x/sys). The Scorecard `Vulnerabilities` check scans repo-wide and is all-or-nothing, so the count never dropped (still 0/10, 21 advisories). ## Known gap This does **not** fix `GO-2025-3512` (`kubevirt.io/csi-driver`): no fixed version exists upstream and Renovate can't even resolve the module (`no-result` in the Dependency Dashboard #2996). Until that one advisory is suppressed via `.osv-scanner.toml` or patched upstream, the Vulnerabilities check stays at 0/10. Tracked separately as a follow-up. ## Validation - `renovate-config-validator`: **Config validated successfully**. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for automated handling of vulnerability alerts. * Enabled updates for indirect Go module dependencies and grouped them together for easier review. * **Chores** * Updated automation settings to label security-related update pull requests appropriately. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
e295f90c78 |
feat(kubernetes): propagate remote-accessible LINSTOR StorageClasses to tenant clusters (#2872)
## What this PR does Tenant Kubernetes clusters previously exposed a single hardcoded `kubevirt` StorageClass mapped to one infrastructure class (`.Values.storageClass`). Users running multiple storage tiers on the management cluster could not reach them from inside tenant clusters (#2094, #1424). This change auto-discovers the management cluster's **remote-accessible LINSTOR** StorageClasses (`provisioner: linstor.csi.linbit.com` with `allowRemoteVolumeAccess != "false"`) via Helm `lookup` and propagates each to tenant clusters under the same name. No Go changes — the kubevirt-csi-driver already supports multiple infra StorageClasses natively. **Behavior** - `packages/apps/kubernetes` `csi.yaml` builds a `storageClasses` map filtered to remote-accessible LINSTOR classes and passes it to the `kubevirt-csi-node` child chart. - `kubevirt-csi-node` renders one tenant StorageClass per entry (`provisioner: csi.kubevirt.io`, `infraStorageClassName: <name>`, `bus: scsi`). - The class named by `.Values.storageClass` (default `replicated`) is marked the tenant default; exactly one default is guaranteed. - The legacy `kubevirt` StorageClass is retained as a non-default alias, so **existing tenant PVCs keep binding with no migration**. - If the `lookup` returns empty (e.g. dry-run), a non-destructive fallback preserves the legacy `replicated` + `kubevirt` mapping — tenant StorageClasses are never wiped on a transient reconcile. **Scope decisions** - **LINSTOR-only**: only StorageClasses cozystack ships and controls are auto-propagated. Node-local classes (e.g. `local`, `allowRemoteVolumeAccess: false`) are excluded because the backing volume cannot follow a live-migrated worker-node VM. Other provisioners (NFS, etc.) are out of scope. - **No enforcement flip**: `infraStorageClassEnforcement` is intentionally left at the current `AllowAll` default. Enabling the allowList would break tenants that have already self-served custom StorageClasses against the management cluster — exactly the users this feature serves. Tightening tenant isolation is left to a separate, discovery-based change. This is a focused, additive alternative to #2095 (kept open), scoped per the LINSTOR-only / no-breakage decisions above. **Upgrade note** If you previously created a tenant StorageClass **named `replicated`** by hand, delete it before upgrading — otherwise the child Helm release hits an ownership conflict adopting the now-managed `replicated` class. ### Tests - Helm unit tests for the fallback map (`kubernetes`) and per-entry rendering + single-default invariant + absence of an enforcement block (`kubevirt-csi-node`; new `Makefile` `test` target so the package is no longer skipped by the runner). - e2e (`run-kubernetes.sh`): asserts `replicated` propagates as the default kubevirt-CSI class, the `kubevirt` alias is retained, and node-local `local` is filtered out. ### Release note ```release-note feat(kubernetes): auto-propagate remote-accessible LINSTOR StorageClasses from the management cluster to tenant Kubernetes clusters under the same name. The class set by `storageClass` (default `replicated`) becomes the tenant default; the legacy `kubevirt` StorageClass is kept as an alias so existing PVCs are unaffected. Node-local classes (e.g. `local`) are not propagated. If you manually created a tenant StorageClass named `replicated`, delete it before upgrading to avoid a Helm ownership conflict. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Remote-accessible LINSTOR `StorageClass` entries now auto-propagate into tenant clusters using the same name. * Tenant-side CSI/KubeVirt `StorageClass` rendering now supports multiple configurable classes with deterministic default selection. * **Improvements** * Enhanced e2e checks to verify `StorageClass` provisioner, parameters, default annotation, and legacy `kubevirt` alias compatibility. * **Tests** * Added Helm unittest suites covering fallback and explicit `storageClass` cases. * Added kubevirt CSI node rendering tests and a `test` target. * **Documentation** * Clarified `storageClass` descriptions and upgrade caveats across charts and schemas. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
ee54a6eb32 |
ci: prune stale and redundant workflows (#2982)
## What this PR does Cleans up four GitHub Actions workflows that are redundant, broken, or noisier than they need to be. Each change is its own commit so they can be reviewed (or reverted) independently. - **`retest.yaml` — removed.** The `/retest` command re-ran a job named `Prepare environment`, but the TIA / job-split refactor of `pull-requests.yaml` collapsed that into a *step* inside the `E2E Tests` job, so the job lookup can never match. Every real `/retest` invocation since fails with `Could not find "Prepare environment" job` — **10 failed runs spanning 2026-01-08 → 2026-06-24, none that ever completed a retest**. (The 41 runs GitHub records as `success` are `issue_comment` events where the job-level `if`-guard was false, so no steps ran.) `gh run rerun --failed` covers the need. - **`codeql.yml` — switch Go to manual build; keep the per-PR run.** The Go `autobuild` ran `make`, whose included `hack/common-envs.mk` does a `$(shell git fetch upstream --tags)` at parse time — a network call to a non-origin remote that has nothing to do with the analysis and is a flake vector. Replace it with `build-mode: manual` that compiles each first-party module explicitly (the root module, `api/apps/v1alpha1`, and the three nested image modules; the gitignored `_repos/*` upstream clones drop out for free on a clean checkout). Same coverage, no upstream fetch. The `pull_request` trigger is **kept on purpose**: per-PR analysis is the only mode that diffs *new* alerts against the base branch and annotates them on the changed lines — the basis for promoting CodeQL to a required pre-merge check (follow-up: add `CodeQL / Analyze (go)` to branch protection and set the code-scanning failure threshold to errors-only). No `paths` filter — a required check that is path-skipped never reports and would block PRs that touch no Go. - **`scorecard.yml` — badge-only; stop polluting code-scanning.** Scorecard uploads one code-scanning alert per check, which floods Security → Code scanning and buries CodeQL's first-party findings. Drop the `upload-sarif` step and the now-unneeded `security-events: write`; keep `publish_results` (the public scorecard.dev badge is the canonical view). Also drop the redundant `push:[main]` trigger — the weekly cron + `branch_protection_rule` already cover it. - **`update-releasenotes.yaml` — path-gate to changelog changes + manual hatch.** The release body is synced from `docs/changelogs/v*.md`, which only lands on main when the changelog PR merges or a maintainer edits an existing changelog. Gate the `push` trigger to `docs/changelogs/v*.md` (the `v*` glob skips `template.md` / `patch-template.md`) so the self-hosted job stops firing on every unrelated main push, and add `workflow_dispatch` as a manual re-sync hatch. - **`SECURITY.md` — document the security scanners.** Add an "Automated security analysis" subsection describing the CodeQL (SAST) and OpenSSF Scorecard (supply-chain) setup, including the manual Go build, the new-alert-on-diff behaviour, the intent to make CodeQL a required pre-merge check, and how findings are dismissed — so the posture is documented, not just encoded in YAML. Not touched: `release-e2e.yaml` — already removed by #2978. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated automation workflows for security checks and release-note generation. * Removed an unused retry workflow and tightened when some checks run. * **Documentation** * Expanded security guidance to better explain automated analysis, alert handling, and reporting. * Clarified when release notes updates are generated. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
da084ef51c |
test(kubernetes): cover fallback-default StorageClass selection (#2872 B1)
Pin the PR #2872 B1 fix: when the management cluster exposes only remote LINSTOR StorageClasses whose names sort after "kubevirt" and none is named the configured storageClass (default "replicated"), the tenant default must be chosen among the propagated classes and never the legacy "kubevirt" alias. helm-unittest cannot reach this branch (no live cluster -> lookup returns empty -> the storageClasses map collapses to the "replicated" fallback), so it is exercised in e2e via a single server-side dry-run render of the kubernetes chart against the live management cluster: two remote LINSTOR classes are added and "replicated" is removed for the one render, then the rendered -csi HelmRelease storageClasses map is asserted to carry exactly one default among the propagated classes, never the "kubevirt" alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
6423cb53e0 |
fix(kubernetes): never auto-select the kubevirt alias as tenant default SC
The legacy "kubevirt" StorageClass alias was inserted into the candidate map before the default was chosen, so the sortAlpha fallback could pick it when propagated class names sort after "kubevirt" (e.g. nvme, ssd) and no configured/replicated class is present. That made the tenant default point at an infraStorageClass that may not exist on the infra cluster, leaving default PVCs Pending. Choose the default from the propagated set before adding the alias. Also quote the templated StorageClass name and infraStorageClassName in kubevirt-csi-node, and document the propagation upgrade caveat, render-time evaluation limitation, and allowRemoteVolumeAccess assumption in the kubernetes README. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
6a1b56f2b5 |
test(kubernetes): cover LINSTOR StorageClass propagation
- csi_test.yaml: assert the fallback storageClasses map yields a default "replicated" plus the non-default "kubevirt" alias, and that the alias tracks the storageClass selector. - kubevirt-csi-node: add a Makefile test target (otherwise the package is silently skipped by the unit-test runner) and storageclasses_test covering per-entry rendering, the single-default invariant, and the absence of any enforcement allowList in driver-config. - run-kubernetes.sh: verify in e2e that "replicated" propagates as the default kubevirt-CSI class, the "kubevirt" alias is retained, and the node-local "local" infra class is filtered out. Refs: #2094 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
021f6141ca |
feat(kubernetes): propagate remote-accessible LINSTOR StorageClasses to tenant clusters
Tenant Kubernetes clusters previously exposed a single hardcoded "kubevirt" StorageClass mapped to one infra class. Discover all remote-accessible LINSTOR infra StorageClasses (provisioner linstor.csi.linbit.com, allowRemoteVolumeAccess != false) via Helm lookup and propagate each to the tenant under the same name. - csi.yaml builds a storageClasses map filtered to remote LINSTOR classes; node-local classes (e.g. "local") are excluded because the backing volume cannot follow a live-migrated worker VM. - The class named by .Values.storageClass (default "replicated") is marked the tenant default; the legacy "kubevirt" StorageClass is kept as a non-default alias so existing tenant PVCs keep binding. - A non-destructive fallback preserves the legacy mapping when the lookup returns empty (dry-run), so tenant StorageClasses are never wiped on a transient reconcile. - infraStorageClassEnforcement is intentionally left disabled (AllowAll) to avoid breaking tenants that already self-served custom StorageClasses under the current default. Refs: #2094, #1424 Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
61f638ea79 | Merge remote-tracking branch 'origin/main' into refactor/build-matrix-2937 | ||
|
Aleksei Sviridkin
|
4752161dc1 |
fix(platform): stamp migrations 45/46 via the cozystack-version helper (#3118)
## What this PR does Migrations 45 and 46 (added by #2931) stamp the `cozystack-version` ConfigMap with an inline `kubectl apply` heredoc instead of sourcing `lib/cozystack-version.sh` and calling `stamp_cozystack_version`, the way migrations 42/43/44 do. The `cozystack-version-stamp.bats` guard forbids any migration ≥42 from bypassing the helper, so `make bats-unit-tests` fails on migration 45. This only surfaces in PR CI (`make unit-tests`) — `main`-push CI does not run the bats suite — so #2931 landed red on this test and now blocks every PR that merges current `main`. This routes both stamps through the helper. The helper renders a byte-identical labeled manifest (golden-pinned in the bats suite), so the change is behaviour-preserving: it just restores the no-label-drift guarantee the heredoc form could silently lose. Verified: `make bats-unit-tests` passes (was failing on migration 45), bash syntax-check clean on both scripts. ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved platform version stamping to keep the version record correctly labeled and consistent during upgrades. * Updated tenant Kubernetes resources to move from v1.30 to v1.31 as part of the migration flow. * **Chores** * Standardized migration behavior by using a shared version-stamping helper for more reliable updates. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
70320e0abf |
fix(platform): stamp migrations 45/46 via the cozystack-version helper
Migrations 45 and 46 (added by #2931) stamp the cozystack-version ConfigMap with an inline `kubectl apply` heredoc instead of sourcing lib/cozystack-version.sh and calling stamp_cozystack_version, the way 42/43/44 do. The cozystack-version-stamp.bats guard forbids any migration >= 42 from bypassing the helper, so `make bats-unit-tests` fails on migration 45. This only surfaces in PR CI (make unit-tests); main-push CI does not run the bats suite, so #2931 landed red on this test and now blocks every PR that merges current main. Route both stamps through the helper. The helper renders a byte-identical labeled manifest (golden-pinned in the bats suite), so this is behaviour- preserving — it just restores the no-label-drift guarantee the heredoc form could silently lose. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
f4a000ef11 |
fix(gateway): pin identical allowedRoutes.kinds on port 443 listeners (cilium#45559) (#3084)
## What this PR does Cilium collapses all listeners that share a port when their `allowedRoutes.kinds` sets differ ([cilium#45559](https://github.com/cilium/cilium/issues/45559)). In the default `TLSPassthroughServices` configuration both HTTPS-terminate listeners (kinds: `[HTTPRoute]`) and TLS-passthrough listeners (kinds: `[TLSRoute]`) land on port 443. The divergent kinds trigger the collapse, which then drops the HTTPRoutes that were accepted by the HTTPS listener. **Fix (Option B — security intact):** give every port-443 listener the same kinds set `[HTTPRoute, TLSRoute]`. - Introduces a shared `port443Kinds` slice in `renderGateway` and assigns it to both `httpsAllowedRoutes.Kinds` and `passthroughAllowed.Kinds`. - GRPCRoute / TCPRoute / UDPRoute are still excluded, so the Layer 7 VAP (`cozystack-route-hostname-policy`) continues to gate exactly the two expected route types. Gateway API itself rejects any HTTPRoute that references a Passthrough `sectionName` at attach time, so the expanded kinds on passthrough listeners add no actual attach surface. - Adds regression test `TestReconcile_Port443ListenersShareKinds` that reconciles a TenantGateway with both an attached HTTPRoute and a TLS-passthrough service, then asserts every port-443 listener carries a non-empty, identical `{HTTPRoute, TLSRoute}` kinds set. - Updates two pre-existing tests (`TestReconcile_TLSPassthroughListenersRendered`, `TestReconcile_HTTPSListenersRestrictRouteKindsToHTTPRoute`) that pinned the old single-kind behavior. ### Release note ```release-note fix(gateway): all port-443 listeners now carry identical allowedRoutes.kinds=[HTTPRoute,TLSRoute] to prevent Cilium (cilium#45559) from collapsing HTTPS-terminate and TLS-passthrough listeners and silently dropping HTTPRoutes ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated port 443 gateway listeners to use a consistent set of allowed route types. * HTTPS (TLS-terminate) and TLS-passthrough listeners now both allow **HTTPRoute** and **TLSRoute**, avoiding configuration discrepancies. * **Tests** * Added regression coverage to verify all port 443 listeners share the same allowed route-kind set and exclude unsupported route types. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Closes #3070 |
||
|
Myasnikov Daniil
|
0b9f5a1005 |
Merge remote-tracking branch 'origin/main' into refactor/build-matrix-2937
Resolve the .github/workflows/pull-requests.yaml conflict: keep this branch's per-package build matrix and carry main's `git diff --binary` patch fix (#e566992) onto every fragment and pr.patch capture, so provider bundles that package image targets regenerate (components*.gz) survive fragment reassembly in finalize and reach the patch e2e applies. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
shreyaabaranwal
|
6acf34aef2 |
test(gateway): pin full port-443 kinds contract across all cert modes (cilium#45559)
Signed-off-by: shreyaabaranwal <shreyabaranwal229@gmail.com> |
||
|
myasnikovdaniil
|
1df8c7c557 |
feat(kubernetes): bootstrap tenant workers with Talos via CABPT (#2931)
## What this PR does > **Supersedes #2610.** Same work, rebased cleanly onto current `main` so it now sits on top of #2867 (the `kube-controller-manager` VAP type-checker fix). That fix is what was killing the base install in #2610's E2E *before* the `kubernetes` app test ever ran, leaving the Talos worker-bootstrap rewrite verified only by code reading. #2610's 67-commit branch is squashed here into 8 logical commits with original authorship preserved (@kvaps and @IvanHunters as authors); the tree is **byte-identical** to #2610's head — no code changes, just the rebase onto `main` (to pick up #2867) plus history cleanup. The detailed review history lives on #2610. Phase 1 of the Kubernetes-app split design (see cozystack/community#8): replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by `cluster-api-bootstrap-provider-talos` (CABPT) and a `clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane pod. Existing tenants keep working — old machines roll out and get replaced by Talos workers without manual intervention. Highlights: - Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the existing kubeadm one in `capi-providers-bootstrap`. - Bump the Kamaji control-plane provider with an upstream-bound patch that exposes `KamajiControlPlane.spec.network.additionalServicePorts`, used to publish trustd (50001/TCP) on the apiserver service. - Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and stable random Talos secrets (`token`, `clusterId`, `clusterSecret`, `bootstrapToken`) using the helm lookup-and-reuse pattern. - Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the tenant `kube-system` via a Helm post-install/upgrade Job. - Render a `TalosConfigTemplate` (worker machineconfig, `generateType: none`) and switch the `MachineDeployment.bootstrap` reference from `KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets and the Kamaji apiserver service being ready. - Boot workers from the Talos openstack image via a CDI `DataVolume` (`source.http.url` → `factory.talos.dev`), expose the system disk as virtio-blk with `blockSize.custom: logical=512, physical=4096` so 4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk (Talos lays out EPHEMERAL itself). - Pin pod/service CIDRs in the worker machineconfig to the tenant ranges (`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap diagnostic against the host pod CIDR. - Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the Kamaji apiserver Service ClusterIP (looked up at render time), so bootstrap survives the chicken-and-egg moment where tenant DNS does not exist yet and the host CoreDNS does not serve the tenant zone. End-to-end on a dev cluster: a fresh tenant brings up a Talos worker, kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving by the talos-csr-signer sidecar), Cilium initialises with hostNetwork against the apiserver ClusterIP, CoreDNS comes up, and the node transitions to `Ready`. ### Release note ```release-note feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes **New Features** * Added Talos worker configuration for `talos.version` and `talos.schematicID` * Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`) for worker remediation tuning * Added image overrides for `kubectl` and `talosCsrSigner` * Exposed additional tenant control-plane Service ports support **Breaking Changes** * Removed Kubernetes v1.30; supported versions start at v1.31 * Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with consolidated Talos system-disk semantics **Removals** * Removed Ubuntu container-disk images and related containerd patch mechanisms * Air-gapped tenant workers temporarily unsupported during the Talos transition **Documentation** * Updated chart docs, parameters, and GitOps upgrade guidance for version bumps <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
133303ec12 |
build(dashboard): pin console image builder to pnpm 11.1.2 and repoint vendor doc link
The console Containerfile ran `corepack prepare pnpm@latest`, so the image build could drift to a newer pnpm major than the `pnpm@11.1.2` pinned in package.json and used by CI. `--frozen-lockfile` guards the lockfile, but pinning the builder makes the image build reproducible and consistent with CI. Also repoints docs/storage-immutability.md off the now-vendored cozystack-ui#6 link to the in-tree console path (follow-up from #2963). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
4838bafbc1 |
fix(dashboard): render error and unknown-type states instead of an infinite spinner
Two console pages ordered their loading guard ahead of the terminal state, making the latter unreachable: - ApplicationDetailPage checked `isLoading || !instance || !ad` before `if (error)`. On a failed GET, isLoading is false and instance stays undefined, so the page spun forever and never showed the not-found message. The error branch now runs first. - ApplicationListPage's `defsLoading || (!ad && plural)` guard stayed true for any unrecognized plural once definitions had loaded, so 'Unknown application type.' was unreachable. The guard now only spins while definitions are loading. Both are pre-existing upstream behaviors carried in by the cozystack-ui vendoring (#2963). Adds co-located regression tests for the failed-GET and unknown-plural paths. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Алексей Артамонов
|
6e093f11bc |
fix(gateway): clean up orphaned ACME account secret on tenant module delete (#3089)
When the gateway tenant module is disabled, cert-manager's ACME account private-key secret cozystack-acme-account was left orphaned (no ownerRef). Add a post-delete cleanup hook that deletes it by name. Verified the secret is referenced only by the gateway module's own cozystack-gateway Issuer, so removing it on gateway teardown is safe. The cleanup Job runs non-root with a read-only root fs and dropped capabilities; RBAC is scoped via resourceNames. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
fad0a3a783 |
docs(security): document CodeQL and Scorecard automated analysis
Record the repository's automated security analysis in SECURITY.md: CodeQL SAST (manual Go build, runs per-PR/push/weekly, new-alert-on-diff, intended as a required pre-merge check, dismiss-with-reason for false positives) and OpenSSF Scorecard (supply-chain posture, published to the scorecard.dev badge, intentionally not uploaded to code scanning). Mirrors the workflow changes in this PR so the security posture is documented, not just encoded in YAML. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
01addfc267 |
build(dashboard): vendor cozystack-ui into the monorepo (#2963)
## What this PR does Moves the `cozystack-ui` console SPA into this monorepo so the UI versions and ships in lockstep with the platform — removing the cross-repo digest-bump PRs and the version drift between the two repositories. The source is vendored at `packages/system/dashboard/images/console/` via `git subtree`, preserving full upstream history. This mirrors the existing in-tree `images/token-proxy/` pattern, and `.helmignore` already excludes `images/*/*` so the source never enters the Helm chart. **Changes** - **Source**: `cozystack-ui` `main` merged under `packages/system/dashboard/images/console/` (history preserved as a merge parent). - **Build**: `packages/system/dashboard/Makefile` builds the console image from the in-tree context instead of cloning the external repo at build time. Dropped `clone-console`/`clean-console` and the `CONSOLE_REPO`/`CONSOLE_BRANCH` variables. The built digest is still pinned into `values.yaml` by `update-values-console` — now in the same commit as the source change rather than a separate follow-up PR. - **CI**: new `.github/workflows/ui-test.yaml` runs pnpm typecheck + unit tests, path-gated to `images/console/**`. The container image continues to build via `make build`. **Verification** - `pnpm install --frozen-lockfile` → `pnpm typecheck` clean → production build produces `dist/`. - 313 unit tests pass. - `helm template` renders; console + token-proxy image references unchanged. **Follow-ups (after merge)** - Archive the standalone `cozystack-ui` repository and redirect contributions here. - Update remaining documentation pointers that reference the old repo path. ### Screenshots N/A — no UI behavior change. This relocates the UI source and rewires the build; the shipped image and digest are unchanged. ### Release note ```release-note build(dashboard): vendor the cozystack-ui console into the monorepo and build it from in-tree source; the shipped image and digest are unchanged for cluster operators ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
b600323df3 |
ci: remove broken retest workflow
The /retest command re-ran a job named 'Prepare environment', but the TIA/job-split refactor of pull-requests.yaml collapsed that into a step inside the E2E Tests job, so the job lookup can never match. Every real /retest invocation since fails with 'Could not find Prepare environment job' — 10 failed runs spanning 2026-01-08 to 2026-06-24, none that ever completed a retest. (The 41 runs GitHub records as 'success' are issue_comment events where the job-level if-guard was false, so no steps ran.) `gh run rerun --failed` covers the need. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
83bfc5fe0a |
ci(scorecard): stop uploading SARIF to code-scanning; drop push trigger
Scorecard posts one code-scanning alert per check, which floods Security > Code scanning and buries CodeQL's first-party findings. The public scorecard.dev badge (publish_results) is the canonical view, so drop the upload-sarif step and the now-unneeded security-events: write. Also drop the push:[main] trigger — the weekly cron and branch_protection_rule already cover it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
71dd0516ef |
ci(codeql): build Go with manual mode instead of autobuild
CodeQL's Go autobuild runs `make` from the repo root, whose included hack/common-envs.mk does a `$(shell git fetch upstream --tags)` at parse time — a network call to a non-origin remote that has nothing to do with the analysis and is a flake vector. Switch to build-mode: manual and compile each first-party module explicitly (the root module, api/apps/v1alpha1, and the three nested image modules); the _repos/* upstream clones are gitignored and absent on a clean checkout, so they drop out for free. Same coverage, no upstream fetch. Keeping the pull_request trigger is deliberate. Per-PR analysis is the only mode that diffs newly introduced alerts against the base branch and annotates them on the changed lines — the basis for promoting CodeQL to a required pre-merge check. No paths filter: a required check that is path-skipped is reported as never-arriving and would block PRs that touch no Go. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
eeab636df0 |
ci(release): run update-releasenotes only on changelog changes
The release body is synced from docs/changelogs/v*.md. That file lands on main only when the changelog PR (tags.yaml generate-changelog) merges, or when a maintainer edits an existing changelog. Path-gate the push trigger to docs/changelogs/v*.md (the v* glob skips template.md / patch-template.md in the same directory) so the self-hosted job stops firing on every unrelated main push and scanning the latest 30 releases for nothing. Also add a workflow_dispatch trigger as a manual re-sync hatch, so a release body that drifts out of sync can be re-synced without touching a changelog file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
a83612bcb5 |
fix(vm-default-images): refresh stale Alpine and openSUSE 16.0 image URLs (#3063)
## What this PR does Refreshes two default golden-image download URLs in `vm-default-images` that had drifted from their canonical source. This came out of a correctness re-check of all 16 default image URLs — the other 14 use floating `current`/`latest` pointers on canonical vendor hosts and need no change. - **alpine-3.21** — was pinned to patch `3.21.6`. Alpine's CDN now ships `3.21.7` and prunes older patches over time, so this hardcoded pin would eventually 404. Bumped to `3.21.7`. (Alpine publishes no `latest` symlink for cloud images, so a patch pin is unavoidable — it just needs to track the newest patch.) - **opensuse-leap-16.0** — pointed at the Open Build Service staging tree (`repositories/openSUSE:/Leap:/16.0:/Images/`) rather than the canonical released-product tree. Switched to `distribution/leap/16.0/appliances/`, which serves the identical image from a more stable, official location. Both new URLs were verified to return HTTP 200 and serve valid QCOW2 images (magic `QFI\xfb`). > Note: `opensuse-leap-15.6` sits on the same OBS staging path, but moving it to the distribution tree would also require switching the image flavor (NoCloud → Minimal-VM Cloud, the only variant published there). That's a behavioral change, so it's intentionally left out of this PR. ### Release note ```release-note fix(vm-default-images): refresh stale Alpine (3.21.6 → 3.21.7) and openSUSE Leap 16.0 default image URLs ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Refreshed two default virtual machine image entries to point to newer download sources. * Updated the openSUSE Leap 16.0 image URL to a new appliance location. * Bumped the Alpine 3.21 image to a newer release version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
d978305e33 |
Merge origin/main into refactor/build-matrix-2937
Resolve conflict in .github/workflows/pull-requests.yaml: keep the new per-package build matrix and fold in main's docker.io->mirror.gcr.io base-image routing (#3042). Apply buildkitd-config: hack/buildkitd.toml to all three new build legs (build, build-talos, finalize) so the mirror fix covers the whole decomposed pipeline, not just the conflicted leg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Алексей Артамонов
|
4b9d67f324 |
fix(kubernetes): clear kamaji datastore-secret finalizer to unblock namespace deletion (#3062)
After deleting a Kubernetes app/tenant, the <release>-datastore-config Secret kept the finalizer finalizer.kamaji.clastix.io/datastore-secret, blocking deletion of the whole namespace. Extend the pre-delete hook to wait for the TenantControlPlane to be removed (so Kamaji can strip the finalizer itself), then, as a backstop, clear the finalizer on the datastore-config Secret if it still lingers. RBAC is scoped via resourceNames to that one Secret (get/patch). The Job runs non-root with a read-only root fs, dropped capabilities and a writable /tmp for the kubectl cache; failures are logged. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Kirill Ilin
|
5cb6ef435f |
feat(keycloak): opt-in separate hostname and route for the admin console (#2954)
## What this PR does
Adds an opt-in way to serve the Keycloak admin console and
Administration
REST API on a separate hostname and route from the public login
endpoints,
so the admin surface can be kept off the public internet.
- New `ingress.adminHost` in the keycloak chart. When set:
- `KC_HOSTNAME_ADMIN` is configured so the admin console generates its
URLs on that hostname;
- the public (login) route is narrowed to the OIDC paths only
(`/realms`, `/resources`, `/.well-known`), so the admin console and
Administration REST API are no longer served on the login hostname;
- a dedicated route (Ingress or HTTPRoute, depending on
`gateway-enabled`)
serves the admin hostname.
- New platform `publishing.ingressNameAdmin`, propagated as
`_cluster.expose-ingress-admin`. The admin route attaches to the
Gateway/ingressClass it names, defaulting to `expose-ingress`. Pointing
it
at a separate (for example, private-IP) Gateway moves admin traffic off
the public one. The separate Gateway itself is provisioned out of band;
this only selects where the admin route attaches.
Everything is empty by default, so existing deployments are unchanged.
Note: `KC_HOSTNAME_ADMIN` on its own is not a security boundary — the
Administration REST API stays reachable on any hostname routed to
Keycloak
(per the Keycloak docs). The path and network separation introduced here
is
what restricts access; full isolation requires attaching the admin route
to
a non-public Gateway.
Covered by helm-unittest suites for the keycloak chart and the platform
value wiring.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(keycloak): add opt-in `ingress.adminHost` to serve the admin console and Administration REST API on a separate hostname/route, plus `publishing.ingressNameAdmin` (`_cluster.expose-ingress-admin`) to attach that route to a separate (e.g. private) Gateway/ingressClass. Disabled by default; existing deployments are unchanged.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional configuration to serve the Keycloak admin console on a
dedicated hostname, with separate routing and ingress/gateway attachment
so admin UI/API can be isolated from public login/OIDC paths.
* Introduced a cluster-level setting to optionally expose admin routes
via a separate ingress/gateway class/namespace.
* **Tests**
* Added new YAML test coverage to validate defaulting, hostname split
behavior, and correct routing for both gateway-enabled and
gateway-disabled modes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
shreyaabaranwal
|
786c098ac6 |
fix(gateway): pin identical allowedRoutes.kinds on port 443 listeners (cilium#45559)
Signed-off-by: shreyaabaranwal <shreyabaranwal229@gmail.com> |
||
|
Aleksei Sviridkin
|
7ff0bf8904 |
ci(build): route PR base-image pulls through mirror.gcr.io (#3042)
## Problem The Pull Request Build job authenticates only to the push registry (OCIR), so every base image (golang, ubuntu, alpine, nginx, …) is pulled from Docker Hub anonymously. Those pulls intermittently fail at `auth.docker.io/token` under Docker Hub's anonymous-pull rate limit, leaving the base image unresolved and failing `make build` — observed across several maintainer-branch runs in a short window. ## Fix Build the PR job on a docker-container buildx builder whose buildkitd routes `docker.io` through `mirror.gcr.io` (Google's public pull-through cache for Docker Hub: no auth, far higher limits, transparent fall-through to Docker Hub on a miss). This reuses the builder pattern already proven in `build-main.yaml` (same `make build`, same `DOCKER_CONFIG` auth forwarding); only the committed `hack/buildkitd.toml` mirror config is new. No base image versions or tags change. ## Risk Switches the PR build from the default embedded `docker` driver to a docker-container builder. `build-main.yaml` already runs the identical `make build` on that driver, so the path is exercised. The "Build Talos image" step uses `docker run`/`skopeo` (not buildx) and is unaffected. ## Fallback If `mirror.gcr.io` is unreachable, BuildKit logs `trying next host` and falls back to `registry-1.docker.io`, so the build still succeeds. Verified locally: a build pointed at a non-resolving mirror fell back to Docker Hub and exited 0 (at a small latency cost from the failed lookups). This strictly reduces build failures without adding a new hard dependency. ## Coverage Covered (the actual failure class — Docker Hub bases): `ubuntu`, `golang*`, `alpine*`, `debian*`, `grafana/grafana`, `nginxinc/nginx-unprivileged`, `clickhouse/*`, `altinity/*`, and explicit `docker.io/library/*`. Not covered, and never the problem: `quay.io/*`, `gcr.io/distroless/static`, `ghcr.io/siderolabs/imager` — the mirror is keyed to `docker.io` only, by design, and these were never subject to Docker Hub limits. ## Optional follow-up (not in this PR) The `moby/buildkit` builder image is still pulled from Docker Hub by the daemon to bootstrap the container builder (the mirror lives inside buildkit, so it cannot mirror its own bootstrap). That is one small pinned image versus dozens of base images, so this change is a strict net reduction in anonymous Docker Hub traffic. Pinning the bootstrap image through a mirror is deliberately left out: a no-fallback mirror on the bootstrap path would add risk for little gain. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved build reliability in pull request checks by using a configured container-based build setup. * Added a safer image-pull path for Docker Hub, with automatic fallback if the mirror is unavailable. * Preserved registry access during image builds so dependency and base-image fetching works more consistently. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
f933918b3f | Merge branch 'main' into ray/build-dockerhub-mirror | ||
|
Aleksei Sviridkin
|
5f9ac28418 |
test(e2e): capture host→pod CNI data-plane on failure (#3047)
## What
On E2E test failure, capture the host→pod CNI data-plane state for each
NotReady pod (scheduled, with a podIP, not terminal) from that pod's
node, into the crust-gather snapshot dir so it ships in the uploaded
report. New helper `hack/e2e-capture-dataplane.sh`, invoked from the
shared on-failure hook in `hack/cozytest.sh` right after the existing
crust-gather collect.
Per affected pod, from the pod's node:
- **host netns** (via the kube-ovn `cni-server`, hostNetwork): `ip route
get <podIP>`, `ip neigh`, `ip rule`, `ip addr show ovn0`, and the kernel
conntrack entries for the podIP (`conntrack -L`, falling back to
`/proc/net/nf_conntrack`).
- **cilium-agent**: `endpoint list`, `bpf ct list global` filtered to
the podIP, a bounded `monitor --type drop`, and `hubble observe
--verdict DROPPED` when the CLI is present.
- **OVS/OVN**: `ovs-ofctl dump-flows br-int`, the OVN `Port_Binding` /
`Logical_Switch_Port` for the pod, and a bounded `ovn-sbctl lflow-list`.
## Why
The install intermittently fails on a host→local-pod data-plane
transient: kubelet on a node hits a local pod's readiness/startup probe
with `connection refused` for several minutes while overlay pod→pod
traffic works, then it self-heals. This stems from the cilium+kube-ovn
chaining config (forced `enable-host-legacy-routing`, CNI
`InstallEndpointRoute:false`), which delegates host→local-pod routing to
kube-ovn/ovn0. Crucially, under `enable-host-legacy-routing` the
host→local-pod path traverses the kernel netfilter stack, not cilium BPF
— so the authoritative evidence is the kernel conntrack table plus the
host route/rule/neighbor state, captured here from the node's host
netns.
The crust-gather snapshot records Kubernetes object state but not the
node's L3 forwarding state, so the mechanism cannot be root-caused after
the fact. This adds exactly that evidence so the next recurrence is
dispositive.
## Diagnostic-only
This changes nothing about install or test behavior — it only collects
more evidence on an already-failed test. It runs after the existing
crust-gather collect and only when a test has already failed, never
mutates the cluster (read-only `get`/`exec`), and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
`|| true`, the whole run is wrapped in a `timeout -k 30 600` wall-clock
backstop, node-global captures are deduped per node, and the pod count
is capped (`MAX_PODS=12`, env-overridable). It no-ops cleanly when no
pods are affected or when a tool/pod is absent. No new retries, no
`EXIT`/`RETURN` traps — consistent with `docs/agents/e2e-testing.md`.
## Validation
`sh -n`, `bash -n`, and `shellcheck` are clean on both touched files.
The affected-pod selection filter and the no-cluster no-op path were
smoke-tested. Not exercised against a live cluster — container/label
names and the kube-ovn LSP naming (`<pod>.<ns>`) are taken from the
`packages/system/{cilium,kubeovn}` charts and the existing
`hack/e2e-cilium-endpoint-leak-healer.sh`; all captures are defensively
`|| true` and time-boxed so absent tooling degrades to a labeled note
rather than a failure.
|
||
|
IvanHunters
|
2cb1b4c665 |
fix(seaweedfs): raise master probe timeouts above the cluster-status backoff (#3068)
## What this PR does The SeaweedFS master's `/cluster/status` HTTP handler calls `topo.Leader()`, which backs off exponentially up to `MaxElapsedTime=20s` while the node has no leader in view. A 3-master cluster bootstrap is a race: a master that briefly becomes the initial raft leader and is then demoted can sit with an empty leader-view for a while, so its `/cluster/status` takes up to ~20s to return (it does return 200 — just slowly). The master liveness and readiness probes both `httpGet /cluster/status:9333` with `timeoutSeconds: 10`, which is shorter than that worst case. Such a master is liveness-killed roughly every ~137s into a permanent CrashLoop (the master uses an `emptyDir` data dir, so each container restart re-reads the same raft state and re-wedges), and its readiness never succeeds — so the master StatefulSet never reaches full readiness and the `seaweedfs-system` HelmRelease install times out. This raises both master probe `timeoutSeconds` to 30 (above the 20s backoff) via the overlay values, deep-merging onto the subchart defaults so only the timeout changes. Both probes must be raised: leaving readiness at 10s would keep the master NotReady on the same slow endpoint even if liveness no longer kills it. The volume component in this same chart already uses a 30s probe timeout, so this aligns master with that precedent. This is purely probe tuning — no raft, protocol, or storage change. The underlying upstream behavior (`/cluster/status` slow for a master with an unhealthy raft view) is tracked upstream and is unchanged across current SeaweedFS releases, so a version bump does not address it. Switching the master to the Hashicorp raft backend is a separate, larger change (it pulls in bootstrap and storage considerations) and is intentionally out of scope here. Operational note: on chart update each master StatefulSet rolls one pod at a time with the new timeouts (raft quorum is preserved); a platform upgrade therefore performs a short graceful roll of master pods on each instance — not a data outage. Adds a helm-unittest pinning both master probe timeouts and guarding that the subchart's other probe fields survive the deep-merge. ### Release note ```release-note fix(seaweedfs): raise the master liveness/readiness probe timeouts above the cluster-status leader-lookup backoff so a master with a transiently empty raft view is not killed into a permanent CrashLoop, which could time out the seaweedfs install ``` |
||
|
IvanHunters
|
cfd9ae30d6 |
fix(mariadb): bind metrics exporter to IPv4 for probe reachability (#3064)
## What this PR does The `mariadb` chart enables a Prometheus `mysqld-exporter` sidecar. The exporter binds the IPv6 wildcard `[::]:9104` by default, while mariadb-operator hardcodes the exporter's liveness/readiness probe as an `httpGet` on port 9104 that kubelet dials over the pod's IPv4 address. On a single-stack-IPv4 data plane the `[::]` listener intermittently refuses the IPv4 connect, so the probe fails roughly one second after the container starts, the pod flaps to NotReady, and the metrics Service keeps the pod IP in `notReadyAddresses` — its endpoint set never populates and Prometheus never scrapes the exporter. This pins the exporter to the IPv4 wildcard via the only chart-side lever the operator exposes for this — `spec.metrics.exporter.args`. The operator appends user-supplied args to its defaults (it does not set `--web.listen-address` itself), so there is no duplicate flag and the DB connection arg is preserved. A `0.0.0.0:9104` socket accepts the kubelet probe deterministically, independent of the exact IPv6 `v6only` submechanism. Conscious tradeoff: this makes the exporter IPv4-only, which is correct for cozystack's single-stack-IPv4 data plane (Cilium runs with `ipv6.enabled=false`). A future IPv6-primary deployment of this app would need `[::]` or both addresses (the exporter's `--web.listen-address` is repeatable in exporter-toolkit ≥ v0.10.0). Adds a helm-unittest pinning the rendered `args`, plus the `test:` Makefile target so CI's chart-test discovery runs it (the chart previously shipped no tests). ### Release note ```release-note fix(mariadb): bind the metrics exporter to IPv4 so its readiness probe is reachable and the metrics endpoint populates reliably ``` |