Record the repository's automated security analysis in SECURITY.md: CodeQL
SAST (manual Go build, runs per-PR/push/weekly, new-alert-on-diff, intended
as a required pre-merge check, dismiss-with-reason for false positives) and
OpenSSF Scorecard (supply-chain posture, published to the scorecard.dev
badge, intentionally not uploaded to code scanning). Mirrors the workflow
changes in this PR so the security posture is documented, not just encoded
in YAML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Moves the `cozystack-ui` console SPA into this monorepo so the UI
versions and ships in lockstep with the platform — removing the
cross-repo digest-bump PRs and the version drift between the two
repositories.
The source is vendored at `packages/system/dashboard/images/console/`
via `git subtree`, preserving full upstream history. This mirrors the
existing in-tree `images/token-proxy/` pattern, and `.helmignore`
already excludes `images/*/*` so the source never enters the Helm chart.
**Changes**
- **Source**: `cozystack-ui` `main` merged under
`packages/system/dashboard/images/console/` (history preserved as a
merge parent).
- **Build**: `packages/system/dashboard/Makefile` builds the console
image from the in-tree context instead of cloning the external repo at
build time. Dropped `clone-console`/`clean-console` and the
`CONSOLE_REPO`/`CONSOLE_BRANCH` variables. The built digest is still
pinned into `values.yaml` by `update-values-console` — now in the same
commit as the source change rather than a separate follow-up PR.
- **CI**: new `.github/workflows/ui-test.yaml` runs pnpm typecheck +
unit tests, path-gated to `images/console/**`. The container image
continues to build via `make build`.
**Verification**
- `pnpm install --frozen-lockfile` → `pnpm typecheck` clean → production
build produces `dist/`.
- 313 unit tests pass.
- `helm template` renders; console + token-proxy image references
unchanged.
**Follow-ups (after merge)**
- Archive the standalone `cozystack-ui` repository and redirect
contributions here.
- Update remaining documentation pointers that reference the old repo
path.
### Screenshots
N/A — no UI behavior change. This relocates the UI source and rewires
the build; the shipped image and digest are unchanged.
### Release note
```release-note
build(dashboard): vendor the cozystack-ui console into the monorepo and build it from in-tree source; the shipped image and digest are unchanged for cluster operators
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The /retest command re-ran a job named 'Prepare environment', but the
TIA/job-split refactor of pull-requests.yaml collapsed that into a step
inside the E2E Tests job, so the job lookup can never match. Every real
/retest invocation since fails with 'Could not find Prepare environment job'
— 10 failed runs spanning 2026-01-08 to 2026-06-24, none that ever completed
a retest. (The 41 runs GitHub records as 'success' are issue_comment events
where the job-level if-guard was false, so no steps ran.) `gh run rerun
--failed` covers the need.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Scorecard posts one code-scanning alert per check, which floods Security >
Code scanning and buries CodeQL's first-party findings. The public
scorecard.dev badge (publish_results) is the canonical view, so drop the
upload-sarif step and the now-unneeded security-events: write. Also drop the
push:[main] trigger — the weekly cron and branch_protection_rule already
cover it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
CodeQL's Go autobuild runs `make` from the repo root, whose included
hack/common-envs.mk does a `$(shell git fetch upstream --tags)` at parse
time — a network call to a non-origin remote that has nothing to do with the
analysis and is a flake vector. Switch to build-mode: manual and compile each
first-party module explicitly (the root module, api/apps/v1alpha1, and the
three nested image modules); the _repos/* upstream clones are gitignored and
absent on a clean checkout, so they drop out for free. Same coverage, no
upstream fetch.
Keeping the pull_request trigger is deliberate. Per-PR analysis is the only
mode that diffs newly introduced alerts against the base branch and annotates
them on the changed lines — the basis for promoting CodeQL to a required
pre-merge check. No paths filter: a required check that is path-skipped is
reported as never-arriving and would block PRs that touch no Go.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The release body is synced from docs/changelogs/v*.md. That file lands on
main only when the changelog PR (tags.yaml generate-changelog) merges, or
when a maintainer edits an existing changelog. Path-gate the push trigger to
docs/changelogs/v*.md (the v* glob skips template.md / patch-template.md in
the same directory) so the self-hosted job stops firing on every unrelated
main push and scanning the latest 30 releases for nothing.
Also add a workflow_dispatch trigger as a manual re-sync hatch, so a release
body that drifts out of sync can be re-synced without touching a changelog
file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Refreshes two default golden-image download URLs in `vm-default-images`
that had drifted from their canonical source. This came out of a
correctness re-check of all 16 default image URLs — the other 14 use
floating `current`/`latest` pointers on canonical vendor hosts and need
no change.
- **alpine-3.21** — was pinned to patch `3.21.6`. Alpine's CDN now ships
`3.21.7` and prunes older patches over time, so this hardcoded pin would
eventually 404. Bumped to `3.21.7`. (Alpine publishes no `latest`
symlink for cloud images, so a patch pin is unavoidable — it just needs
to track the newest patch.)
- **opensuse-leap-16.0** — pointed at the Open Build Service staging
tree (`repositories/openSUSE:/Leap:/16.0:/Images/`) rather than the
canonical released-product tree. Switched to
`distribution/leap/16.0/appliances/`, which serves the identical image
from a more stable, official location.
Both new URLs were verified to return HTTP 200 and serve valid QCOW2
images (magic `QFI\xfb`).
> Note: `opensuse-leap-15.6` sits on the same OBS staging path, but
moving it to the distribution tree would also require switching the
image flavor (NoCloud → Minimal-VM Cloud, the only variant published
there). That's a behavioral change, so it's intentionally left out of
this PR.
### Release note
```release-note
fix(vm-default-images): refresh stale Alpine (3.21.6 → 3.21.7) and openSUSE Leap 16.0 default image URLs
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Updates**
* Refreshed two default virtual machine image entries to point to newer
download sources.
* Updated the openSUSE Leap 16.0 image URL to a new appliance location.
* Bumped the Alpine 3.21 image to a newer release version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The .PHONY: test tweak is out of scope for the PVC cleanup fix; remove it
to keep this PR focused (flagged by review).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
Address review feedback on the post-delete cleanup hook:
- Target Orders/Challenges by cert-manager's own
cert-manager.io/certificate-name=<release>-ui-tls label instead of the
Helm app.kubernetes.io/instance label, which cert-manager never sets on
those resources (the old selector was a silent no-op).
- Bound the Job with activeDeadlineSeconds=120 and backoffLimit=0 so a
stuck finalizer or hung API call cannot block the uninstall.
- Add --wait=false --request-timeout=30s to every kubectl delete/get so
the hook never hangs on slow or finalizer-blocked deletes.
- Surface list failures in the job log instead of redirecting to
/dev/null.
- Tests: assert the bounded execution and security-hardening contract,
pin the rendered document count to 4, and update selector assertions.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
Resolve conflict in .github/workflows/pull-requests.yaml: keep the new
per-package build matrix and fold in main's docker.io->mirror.gcr.io
base-image routing (#3042). Apply buildkitd-config: hack/buildkitd.toml
to all three new build legs (build, build-talos, finalize) so the mirror
fix covers the whole decomposed pipeline, not just the conflicted leg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
After deleting a Kubernetes app/tenant, the <release>-datastore-config
Secret kept the finalizer finalizer.kamaji.clastix.io/datastore-secret,
blocking deletion of the whole namespace. Extend the pre-delete hook to wait
for the TenantControlPlane to be removed (so Kamaji can strip the finalizer
itself), then, as a backstop, clear the finalizer on the datastore-config
Secret if it still lingers. RBAC is scoped via resourceNames to that one
Secret (get/patch). The Job runs non-root with a read-only root fs, dropped
capabilities and a writable /tmp for the kubectl cache; failures are logged.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
The jobservice PVC (resourcePolicy: keep) and cert-manager ACME
http01-solver objects were left orphaned after a Harbor app CR was deleted.
Add a post-delete hook that deletes the release PVCs (selecting by
app.kubernetes.io/instance=<release>-system, the label the upstream subchart
stamps) and the release Certificate/Order/Challenge plus the http01-solver
sweep. A hook is required because resourcePolicy: keep blocks normal reclaim.
The cleanup Job runs non-root with a read-only root fs and dropped
capabilities; failures are logged but never block teardown.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
cert-manager ACME http01-solver Ingress/Pod/Service objects were left
orphaned after a Bucket app CR was deleted (a stuck challenge leaves them
without a live owner to cascade-delete). Add a post-delete hook that deletes
the release Certificate/Order/Challenge and sweeps leftover solver objects by
the acme.cert-manager.io/http01-solver label. RBAC verified against the
vendored cert-manager CRDs. The cleanup Job runs non-root with a read-only
root fs and dropped capabilities; failures are logged but never block teardown.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
## What this PR does
Adds an opt-in way to serve the Keycloak admin console and
Administration
REST API on a separate hostname and route from the public login
endpoints,
so the admin surface can be kept off the public internet.
- New `ingress.adminHost` in the keycloak chart. When set:
- `KC_HOSTNAME_ADMIN` is configured so the admin console generates its
URLs on that hostname;
- the public (login) route is narrowed to the OIDC paths only
(`/realms`, `/resources`, `/.well-known`), so the admin console and
Administration REST API are no longer served on the login hostname;
- a dedicated route (Ingress or HTTPRoute, depending on
`gateway-enabled`)
serves the admin hostname.
- New platform `publishing.ingressNameAdmin`, propagated as
`_cluster.expose-ingress-admin`. The admin route attaches to the
Gateway/ingressClass it names, defaulting to `expose-ingress`. Pointing
it
at a separate (for example, private-IP) Gateway moves admin traffic off
the public one. The separate Gateway itself is provisioned out of band;
this only selects where the admin route attaches.
Everything is empty by default, so existing deployments are unchanged.
Note: `KC_HOSTNAME_ADMIN` on its own is not a security boundary — the
Administration REST API stays reachable on any hostname routed to
Keycloak
(per the Keycloak docs). The path and network separation introduced here
is
what restricts access; full isolation requires attaching the admin route
to
a non-public Gateway.
Covered by helm-unittest suites for the keycloak chart and the platform
value wiring.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(keycloak): add opt-in `ingress.adminHost` to serve the admin console and Administration REST API on a separate hostname/route, plus `publishing.ingressNameAdmin` (`_cluster.expose-ingress-admin`) to attach that route to a separate (e.g. private) Gateway/ingressClass. Disabled by default; existing deployments are unchanged.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added optional configuration to serve the Keycloak admin console on a
dedicated hostname, with separate routing and ingress/gateway attachment
so admin UI/API can be isolated from public login/OIDC paths.
* Introduced a cluster-level setting to optionally expose admin routes
via a separate ingress/gateway class/namespace.
* **Tests**
* Added new YAML test coverage to validate defaulting, hostname split
behavior, and correct routing for both gateway-enabled and
gateway-disabled modes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
The Pull Request Build job authenticates only to the push registry
(OCIR), so every base image (golang, ubuntu, alpine, nginx, …) is pulled
from Docker Hub anonymously. Those pulls intermittently fail at
`auth.docker.io/token` under Docker Hub's anonymous-pull rate limit,
leaving the base image unresolved and failing `make build` — observed
across several maintainer-branch runs in a short window.
## Fix
Build the PR job on a docker-container buildx builder whose buildkitd
routes `docker.io` through `mirror.gcr.io` (Google's public pull-through
cache for Docker Hub: no auth, far higher limits, transparent
fall-through to Docker Hub on a miss). This reuses the builder pattern
already proven in `build-main.yaml` (same `make build`, same
`DOCKER_CONFIG` auth forwarding); only the committed
`hack/buildkitd.toml` mirror config is new. No base image versions or
tags change.
## Risk
Switches the PR build from the default embedded `docker` driver to a
docker-container builder. `build-main.yaml` already runs the identical
`make build` on that driver, so the path is exercised. The "Build Talos
image" step uses `docker run`/`skopeo` (not buildx) and is unaffected.
## Fallback
If `mirror.gcr.io` is unreachable, BuildKit logs `trying next host` and
falls back to `registry-1.docker.io`, so the build still succeeds.
Verified locally: a build pointed at a non-resolving mirror fell back to
Docker Hub and exited 0 (at a small latency cost from the failed
lookups). This strictly reduces build failures without adding a new hard
dependency.
## Coverage
Covered (the actual failure class — Docker Hub bases): `ubuntu`,
`golang*`, `alpine*`, `debian*`, `grafana/grafana`,
`nginxinc/nginx-unprivileged`, `clickhouse/*`, `altinity/*`, and
explicit `docker.io/library/*`. Not covered, and never the problem:
`quay.io/*`, `gcr.io/distroless/static`, `ghcr.io/siderolabs/imager` —
the mirror is keyed to `docker.io` only, by design, and these were never
subject to Docker Hub limits.
## Optional follow-up (not in this PR)
The `moby/buildkit` builder image is still pulled from Docker Hub by the
daemon to bootstrap the container builder (the mirror lives inside
buildkit, so it cannot mirror its own bootstrap). That is one small
pinned image versus dozens of base images, so this change is a strict
net reduction in anonymous Docker Hub traffic. Pinning the bootstrap
image through a mirror is deliberately left out: a no-fallback mirror on
the bootstrap path would add risk for little gain.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved build reliability in pull request checks by using a
configured container-based build setup.
* Added a safer image-pull path for Docker Hub, with automatic fallback
if the mirror is unavailable.
* Preserved registry access during image builds so dependency and
base-image fetching works more consistently.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
On E2E test failure, capture the host→pod CNI data-plane state for each
NotReady pod (scheduled, with a podIP, not terminal) from that pod's
node, into the crust-gather snapshot dir so it ships in the uploaded
report. New helper `hack/e2e-capture-dataplane.sh`, invoked from the
shared on-failure hook in `hack/cozytest.sh` right after the existing
crust-gather collect.
Per affected pod, from the pod's node:
- **host netns** (via the kube-ovn `cni-server`, hostNetwork): `ip route
get <podIP>`, `ip neigh`, `ip rule`, `ip addr show ovn0`, and the kernel
conntrack entries for the podIP (`conntrack -L`, falling back to
`/proc/net/nf_conntrack`).
- **cilium-agent**: `endpoint list`, `bpf ct list global` filtered to
the podIP, a bounded `monitor --type drop`, and `hubble observe
--verdict DROPPED` when the CLI is present.
- **OVS/OVN**: `ovs-ofctl dump-flows br-int`, the OVN `Port_Binding` /
`Logical_Switch_Port` for the pod, and a bounded `ovn-sbctl lflow-list`.
## Why
The install intermittently fails on a host→local-pod data-plane
transient: kubelet on a node hits a local pod's readiness/startup probe
with `connection refused` for several minutes while overlay pod→pod
traffic works, then it self-heals. This stems from the cilium+kube-ovn
chaining config (forced `enable-host-legacy-routing`, CNI
`InstallEndpointRoute:false`), which delegates host→local-pod routing to
kube-ovn/ovn0. Crucially, under `enable-host-legacy-routing` the
host→local-pod path traverses the kernel netfilter stack, not cilium BPF
— so the authoritative evidence is the kernel conntrack table plus the
host route/rule/neighbor state, captured here from the node's host
netns.
The crust-gather snapshot records Kubernetes object state but not the
node's L3 forwarding state, so the mechanism cannot be root-caused after
the fact. This adds exactly that evidence so the next recurrence is
dispositive.
## Diagnostic-only
This changes nothing about install or test behavior — it only collects
more evidence on an already-failed test. It runs after the existing
crust-gather collect and only when a test has already failed, never
mutates the cluster (read-only `get`/`exec`), and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
`|| true`, the whole run is wrapped in a `timeout -k 30 600` wall-clock
backstop, node-global captures are deduped per node, and the pod count
is capped (`MAX_PODS=12`, env-overridable). It no-ops cleanly when no
pods are affected or when a tool/pod is absent. No new retries, no
`EXIT`/`RETURN` traps — consistent with `docs/agents/e2e-testing.md`.
## Validation
`sh -n`, `bash -n`, and `shellcheck` are clean on both touched files.
The affected-pod selection filter and the no-cluster no-op path were
smoke-tested. Not exercised against a live cluster — container/label
names and the kube-ovn LSP naming (`<pod>.<ns>`) are taken from the
`packages/system/{cilium,kubeovn}` charts and the existing
`hack/e2e-cilium-endpoint-leak-healer.sh`; all captures are defensively
`|| true` and time-boxed so absent tooling degrades to a labeled note
rather than a failure.
Add the operator-facing knob that drives the new exposure model. When
publishing.exposureClass.name is set, the platform renders a cluster-scoped
ExposureClass binding the chosen backend (externalIPs, metallb, cilium,
robotlb) and VIP address scope, and carries the class name through the
cozystack-values Secret. The host ingress then renders a ServiceExposure
for its ingress-nginx Service, which the cozystack-controller translates
into the backend's pool and announcer.
The default (empty name) renders neither the ExposureClass nor the
ServiceExposure, so clusters on the externalIPs path are unchanged on
upgrade. The ingress Service shape is untouched: it is already published
as type LoadBalancer whenever externalIPs is empty.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add the serviceexposure controller to the cozystack-controller binary. It
resolves a ServiceExposure's ExposureClass to a backend and renders the
backend-specific pool and announcer CRs, then reports the assigned IPs in
status.
Backends are pluggable behind a small interface, one per file:
- externalIPs: renders nothing (the node owns the address) — preserves the
historical default behaviour.
- metallb: IPAddressPool + L2Advertisement in cozy-metallb, scoped to the
target Service's namespace.
- cilium: CiliumLoadBalancerIPPool (cilium.io/v2) + CiliumL2AnnouncementPolicy
(cilium.io/v2alpha1), scoped to the Service via serviceSelector.
- robotlb: renders nothing (the cloud LB owns allocation and announcement).
The rendered CRs belong to foreign API groups not in the Go scheme, so they
are emitted as unstructured objects. They carry no OwnerReference (the
Cilium CRs are cluster-scoped and the MetalLB CRs live in a different
namespace than the owning ServiceExposure), so ownership is tracked via a
management label plus owner annotations, cleanup is driven by a finalizer,
and the controller refuses to take over an identically-named resource it
does not manage. The controller never mutates the target Service, leaving
its shape to the chart that renders it.
Grant the controller write access to metallb.io and cilium.io pool /
announcer resources.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
The SeaweedFS master's `/cluster/status` HTTP handler calls
`topo.Leader()`, which backs off exponentially up to
`MaxElapsedTime=20s` while the node has no leader in view. A 3-master
cluster bootstrap is a race: a master that briefly becomes the initial
raft leader and is then demoted can sit with an empty leader-view for a
while, so its `/cluster/status` takes up to ~20s to return (it does
return 200 — just slowly). The master liveness and readiness probes both
`httpGet /cluster/status:9333` with `timeoutSeconds: 10`, which is
shorter than that worst case. Such a master is liveness-killed roughly
every ~137s into a permanent CrashLoop (the master uses an `emptyDir`
data dir, so each container restart re-reads the same raft state and
re-wedges), and its readiness never succeeds — so the master StatefulSet
never reaches full readiness and the `seaweedfs-system` HelmRelease
install times out.
This raises both master probe `timeoutSeconds` to 30 (above the 20s
backoff) via the overlay values, deep-merging onto the subchart defaults
so only the timeout changes. Both probes must be raised: leaving
readiness at 10s would keep the master NotReady on the same slow
endpoint even if liveness no longer kills it. The volume component in
this same chart already uses a 30s probe timeout, so this aligns master
with that precedent. This is purely probe tuning — no raft, protocol, or
storage change.
The underlying upstream behavior (`/cluster/status` slow for a master
with an unhealthy raft view) is tracked upstream and is unchanged across
current SeaweedFS releases, so a version bump does not address it.
Switching the master to the Hashicorp raft backend is a separate, larger
change (it pulls in bootstrap and storage considerations) and is
intentionally out of scope here.
Operational note: on chart update each master StatefulSet rolls one pod
at a time with the new timeouts (raft quorum is preserved); a platform
upgrade therefore performs a short graceful roll of master pods on each
instance — not a data outage.
Adds a helm-unittest pinning both master probe timeouts and guarding
that the subchart's other probe fields survive the deep-merge.
### Release note
```release-note
fix(seaweedfs): raise the master liveness/readiness probe timeouts above the cluster-status leader-lookup backoff so a master with a transiently empty raft view is not killed into a permanent CrashLoop, which could time out the seaweedfs install
```
## What this PR does
The `mariadb` chart enables a Prometheus `mysqld-exporter` sidecar. The
exporter binds the IPv6 wildcard `[::]:9104` by default, while
mariadb-operator hardcodes the exporter's liveness/readiness probe as an
`httpGet` on port 9104 that kubelet dials over the pod's IPv4 address.
On a single-stack-IPv4 data plane the `[::]` listener intermittently
refuses the IPv4 connect, so the probe fails roughly one second after
the container starts, the pod flaps to NotReady, and the metrics Service
keeps the pod IP in `notReadyAddresses` — its endpoint set never
populates and Prometheus never scrapes the exporter.
This pins the exporter to the IPv4 wildcard via the only chart-side
lever the operator exposes for this — `spec.metrics.exporter.args`. The
operator appends user-supplied args to its defaults (it does not set
`--web.listen-address` itself), so there is no duplicate flag and the DB
connection arg is preserved. A `0.0.0.0:9104` socket accepts the kubelet
probe deterministically, independent of the exact IPv6 `v6only`
submechanism.
Conscious tradeoff: this makes the exporter IPv4-only, which is correct
for cozystack's single-stack-IPv4 data plane (Cilium runs with
`ipv6.enabled=false`). A future IPv6-primary deployment of this app
would need `[::]` or both addresses (the exporter's
`--web.listen-address` is repeatable in exporter-toolkit ≥ v0.10.0).
Adds a helm-unittest pinning the rendered `args`, plus the `test:`
Makefile target so CI's chart-test discovery runs it (the chart
previously shipped no tests).
### Release note
```release-note
fix(mariadb): bind the metrics exporter to IPv4 so its readiness probe is reachable and the metrics endpoint populates reliably
```
## What
Raise the `kafka` and `zookeeper` components in the Kafka e2e test from
the `nano` preset (250m / 128Mi) to `c1.small` (1 CPU / 1Gi), the chart
default.
## Why
128Mi cannot hold the Strimzi Kafka 3.9.1 and ZooKeeper JVMs. The
brokers OOMKill on startup (exit 137) and enter CrashLoopBackOff, so the
pods never reach Ready, the `Kafka` CR and its HelmRelease never go
Ready, and the 5m readiness wait times out. It presents as a flake
rather than a hard failure because 128Mi sits right at the JVM-start
threshold — occasionally a broker becomes Ready before its RSS crosses
the cgroup limit.
`c1.small` (1Gi) comfortably fits both JVMs and matches the chart
default, so the test exercises the documented default sizing. The e2e
sandbox VMs are 8 CPU / 24Gi, so 1Gi per pod (4Gi total) across the four
pods (kafka-0/1, zookeeper-0/1) is well within budget.
## Scope
Test-only; values changed, test structure untouched. MariaDB's e2e test
also uses `nano` but was deliberately left as-is: it runs MariaDB (C) +
a Go exporter with no JVM, its chart default is already `t1.nano`
(128Mi), and it shares the `nano` preset with the passing
postgres/redis/qdrant tests — not the same OOM class.
## What
Add a bounded escalation tier to the E2E cilium endpoint-leak healer:
when a pod stays wedged on `FailedCreatePodSandBox` "IP ipv4:<X> is
already in use" for the same IP across several heal cycles, restart the
cilium-agent on that node. Keep the existing per-endpoint remedies as
the first tier.
## Why
The leak (cilium/cilium#38313 class) sometimes lives purely in the
cilium-agent's in-memory endpoint state. The existing remedies —
`cilium-dbg endpoint disconnect` and delete-rescheduling the wedged pod
— cannot reach it: IPAM keeps re-handing the SAME released IP, so the
recreated pod re-wedges on it, and on some variants the stale entry is
not even in the queryable endpoint registry, so disconnect is a no-op.
Both are inescapable loops that fail the install. The only thing that
clears an in-memory leak is restarting the agent, which drops the node's
endpoint registry and rebuilds it from CRDs/IPAM without the stale
entry.
## How
- Track recurrence per `(node, ip)`. After `ESCALATE_AFTER` (default 3)
heal cycles of a futile first-tier remedy on the same IP, restart that
node's cilium-agent pod.
- Bounded by `MAX_ESCALATIONS_PER_NODE` (default 2); after the cap, log
loudly and surface the failure instead of restarting again. State is
per-node and resets after each restart so one restart covers every
leaked IP on the node.
- A genuine duplicate-IP (endpoint backs a live owner) is refused before
counting, so an agent restart can never mask a real product bug.
- RBAC unchanged: the existing cluster-wide `pods` `get/list/delete`
already covers deleting a cilium-agent pod in `cozy-cilium`.
E2E-test-only resilience; no product code change. Remove alongside the
rest of the healer once a fixed Cilium ships.
## What this PR does
**Problem.** `ovn-controller` runs inside the `ovs-ovn` DaemonSet's
single `openvswitch` container, sharing one CPU limit with
`ovsdb-server` and `ovs-vswitchd`. The container CPU limit was `2` cores
and the request `10m`. For three latency-sensitive daemons on a busy
cluster that ceiling is tight: under a burst of pod creation (a large
install with many concurrent HelmReleases) `ovn-controller` is CPU-bound
while programming OpenFlow for newly created logical ports, and a `10m`
request leaves it almost no guaranteed share under node CPU contention.
**Change.** Raise the `ovs-ovn` container CPU limit from `2` to `4`
cores and the request from `10m` to `100m`, so the co-located daemons
have headroom under burst and `ovn-controller` keeps a guaranteed
scheduling share under contention. Memory is unchanged — the change is
scoped to CPU.
**Scope — hardening, not a proven fix.** This change started from a
hypothesis that CPU throttling was lengthening flow-programming and
causing transient host→pod unreachability (`connection refused` from a
pod's own node) during burst installs. A live trace of a captured
occurrence of that symptom showed **zero** `ovs-ovn` CPU throttling in
the failure window, so the throttle ceiling was **not** the cause of
that failure. This PR therefore lands as a cheap, safe, well-bounded
headroom improvement on a ceiling that is genuinely tight for three
co-located daemons — not as a fix for the unreachability flake, whose
root cause is separate and still open.
**Test.** `helm template` confirms the rendered `ovs-ovn` DaemonSet now
sets `requests.cpu: 100m` and `limits.cpu: 4` (with `ephemeral-storage:
1Gi` inherited from the subchart default). A new helm-unittest
regression guard pins the raised limit and request.
### Screenshots
No UI changes.
### Release note
```release-note
fix(kube-ovn): raise the ovs-ovn container CPU limit to 4 cores (request to 100m) so the co-located ovn-controller/ovsdb-server/ovs-vswitchd daemons have CPU headroom under burst installs
```
## What this PR does
**Problem**
On a loaded install the `monitoring-agents` fluent-bit DaemonSet
crash-loops and its HelmRelease times out (`timeout waiting for:
[DaemonSet/.../monitoring-agents-fluent-bit status: 'InProgress']`),
cascading into velero → backupstrategy-controller `DependencyNotReady`
and failing the whole install.
Root cause: fluent-bit's HTTP server on `:2020` (the liveness/readiness
target) binds only AFTER the `filter:kubernetes` "testing connectivity
with API server" step. During a concurrent install the kube-API can be
unreachable for ~10s (observed `connection #144 timeout after 10 seconds
to: kubernetes.default.svc:443`), which delays the `:2020` bind. The
vendored subchart's default probes (`initialDelaySeconds 0,
timeoutSeconds 1, failureThreshold 3`) fire into a connection-refused
port immediately, and the single-threaded HTTP server can't always meet
a 1s timeout while the engine is busy — so liveness SIGTERMs fluent-bit
(exitCode 0, not OOM) into a restart loop.
**Fix**
Relax both probes in the first-party umbrella `values.yaml`
`fluent-bit:` block: `initialDelaySeconds 15, periodSeconds 10,
timeoutSeconds 5, failureThreshold 6` (~75s tolerance from container
start, ~60s steady-state). The `httpGet` target is unchanged — path `/`
on port `http` (`:2020`). The change is values-only; the vendored
subchart is not touched.
**Why load-triggered (not always-broken)**
A steady-state cluster runs the identical probe config with the
DaemonSet 3/3 Ready, 0 restarts, 26 days. The probe only fails when the
`:2020` bind is delayed by API unreachability under install-time load.
Steady-state-stable + flaky-only-under-load = probe-too-aggressive, not
a target or config error.
**Test**
Added a helm-unittest (`tests/fluent-bit_probes_test.yaml`) pinning both
the relaxed timing and the preserved `:2020` httpGet target (path `/`,
port `http`) on the rendered DaemonSet, so a future change that
re-tightens the probes or breaks the path/port is caught.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(monitoring-agents): relax fluent-bit liveness/readiness probes so the DaemonSet no longer crash-loops on a loaded install when its :2020 HTTP server binds late behind a busy kube-API
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Adjusted Fluent Bit health probes to use more relaxed timing, reducing
the chance of restart loops on busy installations.
* Kept the readiness check pointed at the expected HTTP path and port
while making probe behavior more stable.
* **Tests**
* Added test coverage to verify the generated probe configuration stays
consistent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Under node CPU contention the host `linstor-controller` JVM cannot
answer its `GET /health` liveness probe within the piraeus-operator
default timing (`timeoutSeconds: 1`, `failureThreshold: 3`). The kubelet
then liveness-kills the container (exit 137, reason `Error`, not
`OOMKilled`). The resulting crash-loop leaves the `linstor-controller`
Service with no ready endpoints, `linstor-csi` gets `EPERM`, and
RWX-Filesystem `DataVolume`s never bind — breaking tenant Kubernetes /
NFS provisioning.
The operator-generated controller pod sets no `resources` (so no CPU
request and no scheduling/CPU guarantee) and ships the tight default
probe. This change, on the `linstor-controller` container in
`spec.controller.podTemplate`:
- adds `resources.requests` (cpu `250m`, memory `700Mi`) — requests
only, no limits: a CPU request guarantees CPU shares so the JVM keeps
answering `/health` under contention, while omitting limits avoids
throttling the JVM or OOM-killing it. The memory value is a conservative
starting estimate; tune it to observed RSS.
- relaxes the liveness timing to `timeoutSeconds: 5` and
`failureThreshold: 6` so a JVM GC pause or CPU contention does not trip
a liveness kill.
piraeus-operator merges `spec.controller.podTemplate` into the generated
Deployment as a strategic merge patch, so the liveness override carries
only the two timing fields; the operator's `httpGet` target (`GET
/health` on the api port) is left intact to avoid pinning a wrong path
or scheme. Both knobs are exposed under `controller.*` in `values.yaml`
for tuning.
This is a latent chart weakness amplified by load, not a regression: on
the same branch, image, and chart, a green run had the controller stable
at `restartCount=0`, while a CPU-loaded run on identical code
crash-looped. Same code, different outcome under load.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(linstor): give the linstor-controller CPU/memory requests and a more tolerant liveness probe so it no longer crash-loops under node CPU contention, which previously left the controller Service with no endpoints and blocked RWX volume provisioning
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added default controller resource requests for the Linstor controller.
* Added configurable liveness probe timing settings for the controller
container.
* **Tests**
* Added coverage to verify the controller pod template applies the
expected resource requests and liveness probe timing values.
* Confirmed the controller image remains correctly pinned while the new
settings are applied.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.
## Fix
Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.
## Why
The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.
## Test
- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.
* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
**Problem**
opensearch-operator installs intermittently fail with Helm reporting the
controller-manager `Deployment status: 'Failed'`. The real driver is the
manager container (`operator-controller-manager`) crashlooping on
`leader election lost`: controller-runtime calls `os.Exit(1)` when it
misses a lease-renewal deadline under the apiserver/etcd latency that
install-time load creates. The Deployment runs a single replica
(`replicas: 1`) yet still passes `--leader-elect`, so leader election
adds no availability — its only effect is to let a transient
control-plane blip self-terminate the sole manager. The kube-rbac-proxy
`BackOff` seen alongside is collateral restart-window noise, not the
cause: on a long-running cluster the proxy shows `restartCount: 0` while
the manager has restarted repeatedly with `exitCode: 1`, `reason:
Error`.
**Fix**
Gate `--leader-elect` on the replica count instead of passing it
unconditionally. A new `manager.replicaCount` value (default `1`) drives
both `spec.replicas` and the flag: at one replica the manager runs
without leader election and rides out control-plane latency instead of
crashlooping; setting `replicaCount > 1` re-enables `--leader-elect`
automatically so a genuine HA deployment still coordinates. Because both
derive from one value, there is no window where multiple managers run
without a lease.
The same single-replica path also sets the Deployment strategy to
`Recreate`. With leader election off, a default `RollingUpdate` would
briefly run two managers (maxSurge) during an operator upgrade, both
reconciling without lease coordination; `Recreate` closes that window at
zero availability cost on one replica. Installs scaled past one replica
keep `RollingUpdate`.
The arg is hardcoded in the vendored opensearch-operator 2.8.0 chart, so
the change ships as
`packages/system/opensearch-operator/patches/leaderElection.diff`,
re-applied by the package `make update` target after `helm pull
--untar`. This mirrors the existing fluxcd-operator patch mechanism so
the customization survives a chart refresh. A helm-unittest suite pins
both states: the default render drops `--leader-elect` and sets
`strategy: Recreate`; a `replicaCount: 2` render restores
`--leader-elect` and leaves the strategy at the RollingUpdate default.
### Screenshots
Not applicable — no UI change.
### Release note
```release-note
fix(opensearch-operator): disable leader election on the single-replica operator Deployment so a transient apiserver/etcd latency spike no longer crashloops the manager at install time; leader election re-enables automatically when scaled past one replica
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable replica count for the operator manager, with a
default of 1.
* Single-replica deployments now use a safer rollout strategy and skip
leader election.
* Multi-replica deployments automatically enable leader election.
* **Tests**
* Added coverage for single- and multi-replica deployment behavior.
* **Documentation**
* Updated configuration notes to explain replica count behavior and
related deployment changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
**Problem**
The `foundationdb` e2e test fails intermittently with a `.status`
jsonpath poll timeout even though the FoundationDB cluster is fully
healthy and reconciled. The `generations.reconciled`,
`hasListenIPsForAllPods`, `health.fullReplication` and `health.healthy`
gates polled their status fields with a 60s ceiling, while the
strictly-earlier `health.available` gate already allowed 300s. The
operator stamps `generations.reconciled` only after every process holds
its role for `minimumUptimeSecondsForBounce` (60s, set in
`packages/apps/foundationdb/templates/cluster.yaml`), which makes
`reconciled` — and the health fields gated on it — structurally the last
status fields to converge. The short ceiling closed the gate seconds
before the field landed; a snapshot taken a few seconds after a failure
shows the live object fully reconciled (`generations.reconciled: 1`,
`health.healthy: true`).
**Fix**
Raise those four late-converging gates to 300s, matching the existing
`health.available` budget. The waits stay event-driven (`until …; do
sleep …; done`) — only the ceilings widen, no new retries or fixed
sleeps. A comment records why `reconciled` converges last so the budget
ordering stays legible to future readers. The early-converging 60s gates
(`configured`, `connectionString`, `databaseConfiguration.*`,
`desiredProcessGroups`) and the spec-field gates are left untouched,
since they settle before `health.available` and 60s is already adequate.
**Why not just rerun**
Rerunning masks a deterministic ordering bug: the latest-converging
field carried the tightest budget. The cluster was healthy on every
"failure" — the test was racing its own success criteria, so the failure
recurs until the budget ordering is corrected. This change touches only
the test harness; the platform behaviour it asserts is unchanged.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased waiting time in one end-to-end FoundationDB setup test so
cluster status checks have more time to stabilize.
* Reduced flaky failures by allowing key health and reconciliation
fields to converge before the test continues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
On a fresh install, the install step can fail with:
```text
Error creating resource via patch ... apps.cozystack.io/v1alpha1 Kind=Bucket
name=cozy-backups namespace=tenant-root: namespaces "tenant-root" not found
```
`backupstrategy-controller` renders the `cozy-backups` Bucket **into**
the `tenant-root` namespace, but its `dependsOn` only guards the CRDs it
needs (backup-controller / bucket-application / objectstorage-controller
/ velero) — not the existence of the target namespace. `tenant-root` is
provisioned by `cozystack-basics` (the static tenant-root release), and
there is no ordering edge to it. On an unlucky Flux reconcile order the
Bucket patch races namespace creation, fails, and that transient failure
races the install script's `kubectl wait` deadline → the run fails.
This is latent and not introduced by any particular change — the
Bucket-into-tenant-root predates recent branches — so it flakes installs
on changes that have nothing to do with backups.
## Fix
Add `cozystack.cozystack-basics` to `backupstrategy-controller`'s
`dependsOn` so the `tenant-root` namespace exists before the Bucket is
applied. `cozystack-basics` is a dependency-graph leaf (nothing depends
on it), so no cycle is introduced.
Pinned with a helm-unittest regression guard
(`tests/sources_backupstrategy_dependson_test.yaml`) asserting the edge
— mirroring the existing `sources_basics_dependson_test.yaml` pattern.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved install ordering so the backup strategy component waits for
required foundational services before deploying, reducing race
conditions during fresh installs.
* Clarified dependency relationships for cluster-wide platform
components to better reflect safe upgrade and installation behavior.
* **Tests**
* Added regression coverage to verify the backup strategy package
declares all required dependencies, including namespace and CRD
providers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Problem
The `Bootstrap Talos cluster` E2E step fails intermittently with `❌ Test
failed: Bootstrap Talos cluster (exit 1)` even though the cluster is
healthy. The exit comes from the etcd-members convergence gate, which is
bounded at `timeout 180`.
A 3-node etcd converges via Talos's serialized learner promotion — etcd
admits one learner at a time, so the third member is only promoted after
the second has fully caught up. On a loaded ephemeral runner this
deterministic path can legitimately approach ~3 minutes of wall clock
and photo-finish against the 180s ceiling, timing out a cluster that has
in fact converged (every member reaches `Health check successful`; no
kernel panic / OOM / boot failure / network-unreachable in dmesg).
Observed: a run where the third member reached etcd health at +178.6s
against the 180s budget, failing ~1.4s short.
## Fix
Widen the etcd-members convergence budget from 180s to 300s. The wait is
already event-driven (`until talosctl etcd members …; do sleep 1; done`)
— only the ceiling was too tight. No retry is added and the
dmesg-on-failure diagnostic is preserved.
300s matches the repo's other in-line budgets for deterministic
convergence (HelmRelease-ready 5m, harbor/NFS 10m, install 15m).
## Why not just rerun
This is not environmental noise: the cluster converges correctly every
time, it is only timed out. Rerunning wastes a full ~1h E2E cycle to
re-roll the dice on whether the runner shaves the last ~2s. Correcting
the mistuned budget removes the photo-finish.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Extended the cluster bootstrap wait time for control plane health
checks, reducing false failures during slow startup.
* Improved readiness handling so cluster setup is more resilient before
moving on to later initialization steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The StatefulSet data PVC (qdrant-storage-<release>-0) was left orphaned
after a Qdrant app CR was deleted. The vendored upstream chart does not
support persistentVolumeClaimRetentionPolicy, so add a post-delete PVC
cleanup hook (modeled on the MariaDB hook) and pass persistence.additionalLabels
through the HelmRelease so the data PVC carries a release-unique
app.kubernetes.io/instance label the hook selects on. The cleanup Job runs
non-root with a read-only root fs and dropped capabilities; failures are
logged but never block namespace teardown.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
The mariadb-operator creates <release>-metrics-password and
<release>-repl-password Secrets as a side effect of metrics/replication
being enabled. These are not Helm-owned and survive `helm uninstall`,
leaving orphaned Secrets in the tenant namespace after the app CR is
deleted.
Extend the existing post-delete cleanup hook to sweep them: grant the
cleanup Role RBAC on secrets and delete both password Secrets by name.
Add a helm-unittest suite locking in the behaviour (and a `test` Makefile
target so the package's tests run in CI).
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
After deleting an OpenBAO app CR, the StatefulSet data PVC (data-<release>-0)
was left orphaned in the tenant namespace, consuming storage permanently.
StatefulSet-templated PVCs are never garbage-collected on StatefulSet deletion
by default, and the chart set no persistentVolumeClaimRetentionPolicy.
Pass a retention policy through the HelmRelease values. The vendored upstream
chart renders it onto the StatefulSet from
server.persistentVolumeClaimRetentionPolicy (not server.dataStorage.*):
whenDeleted: Delete reclaims the data PVC on release deletion, while
whenScaled: Retain keeps a member's data PVC on HA scale-down.
Also add a 'test' Makefile target so the package's helm-unittest suite runs in
CI (hack/helm-unit-tests.sh only runs packages that define a 'test' target).
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Introduce a new network.cozystack.io/v1alpha1 API group with two kinds
that form the mechanism-agnostic LoadBalancer exposure model:
- ExposureClass (cluster-scoped): the admin binds a logical class to a
concrete backend (externalIPs, metallb, cilium, robotlb) and its VIP
address scope, following the StorageClass / IngressClass pattern.
- ServiceExposure (namespaced): the tenant-facing intent to expose a
Service externally via a named class, with status reporting the
resolved backend and assigned IPs.
Wire api/network into the controller-gen CRD generation so deepcopy and
the two CRD manifests are produced alongside the existing groups.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrite the SecurityGroup e2e for the membership model: assert the backing
CiliumNetworkPolicy's endpointSelector is the group's own membership label,
that attachments ride a storage-owned annotation hidden from the view, and
that fromApp/fromSG peers project to lineage- and membership-label endpoint
selectors the real cilium.io/v2 CRD accepts. A second test creates a
managed-labeled pod and asserts the securitygroup-controller stamps the
membership label on attach and clears it on detach.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrite the SecurityGroup design for the membership model: each group's own
identity label, attachments stored in a backing-policy annotation, the
REST/controller split, live fromSG/toSG references, the controller's hardening
invariants and the eventual-consistency window contract. Spell out that the
default-deny baseline flip and the membership admission webhook are out of
scope and why, and that membership relocates rather than fixes the "tenant
firewalls its own managed application" concern.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add the securitygroup-controller Helm package (Deployment, ServiceAccount,
ClusterRole and binding, image build) and register it as a default system
package so it ships with the platform.
The ClusterRole grants cluster-wide pod patch — the controller stamps the
SecurityGroup membership label on managed-app pods across dynamically-created
tenant namespaces, so the grant cannot be namespace-scoped — plus read and
finalizer patch on the backing CiliumNetworkPolicies, and leader-election
leases. Tenants gain no new permissions; the membership label is written
exclusively by this platform controller. helm-unittest pins the pod-patch and
ciliumnetworkpolicies grants so a dropped verb fails the suite rather than
silently failing closed at runtime.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a controller that keeps each SecurityGroup's membership label
(securitygroup.sdn.cozystack.io/<name>) in sync with its attachments: it
stamps the label onto the pods of every attached application and removes it
when an application is detached or the SecurityGroup is deleted. The backing
CiliumNetworkPolicy's endpointSelector matches that label, so this is what
makes a SecurityGroup apply to its members.
The controller watches the marked backing policies and managed-app pods. It
resolves each attachment through the lineage labels the lineage webhook
already stamps — never the attachment list directly — and only ever labels
pods in the SecurityGroup's own namespace, using single-key merge patches so
it cannot clobber another group's label or a pod's lineage labels. A finalizer
on the backing policy guarantees the labels are stripped before the policy is
removed. Informer caches are bounded to managed pods and SecurityGroup-owned
policies.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dataplane diagnostic's announcer-node detection reported the wrong
node in two failover situations and could match the wrong IP:
- A node that announced an IP and then withdrew it (with no
re-announce) was still reported as the announcer, because a withdraw
did not retract a prior announce.
- Speaker logs are concatenated per-pod, not globally time-sorted, so
a stale former owner's announce could outrank the true current
owner purely by its position in the input.
- The IP was matched as a substring, so a query for 192.0.2.5 also
matched a 192.0.2.50 line.
Reimplement the detection to report a node only when that node's LAST
own announce/withdraw event for the exact IP is an announce, which is
robust to both concat order and L2 failover, and anchor the IP match to
a whole IP-literal token instead of a substring.
Also fix the heavy-capture budget: the MAX_LBS cap counted every
enumerated LB, so a broken LB enumerated after a cap's worth of
reachable LBs was never captured. Count only LBs actually captured, via
a small pure budget helper, so reachable/skipped LBs do not consume the
budget. Document that this cap bounds work, not wall-clock -- at two or
more unreachable LBs the real wall-clock bound is the outer backstop at
the call site.
Add bats coverage pinning the fixed announcer behavior (withdraw,
concat-order, exact-IP) and the captured-budget accounting.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The on-failure dataplane diagnostic only captured host->pod CNI state for
NotReady pods. A recurrent flake presents differently: a tenant Service
type=LoadBalancer IP goes unreachable while its backend pod/VMI stays Ready
and Running, so there is no NotReady pod and the hook captured nothing --
leaving the host->cross-node->bridged-worker-VMI->nodePort path
uncharacterised, which is exactly where root-causing has stalled.
Add a gated LoadBalancer-datapath section. It enumerates LoadBalancer
services with an ingress IP, derives the EndpointSlice backend, the
nodePort/externalTrafficPolicy, and the announcer node from the hostNetwork
MetalLB speaker logs, then probes the LB IP from a host netns. Only an LB
whose every probe fails gets the heavy capture, and that probe doubles as
the traffic for the tcpdumps. For a failing LB it captures, on both the
announcer and endpoint nodes, the host cilium LB map, kernel
conntrack/neigh, br-int flows, ovn-controller timing, and a live tcpdump on
the geneve tunnel and the backend tap -- localising the failing hop to
host-cilium programming vs kube-ovn delivery. In-guest VMI capture is
documented as a deferred stretch that needs tenant-cluster access.
The existing NotReady-pod capture is unchanged. The new section is
best-effort and time-boxed like the rest of the script, and a unit test
pins the pure enumeration, announcer-node, and capture-or-skip logic against
mock input.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Give each SecurityGroup its own membership label
(securitygroup.sdn.cozystack.io/<name>) and select it from the backing
CiliumNetworkPolicy, instead of deriving the endpointSelector from a single
app's lineage labels. spec.targetRef becomes spec.attachments, so one
SecurityGroup can apply to several applications at once.
Peers move from free-form fromEndpoints/toEndpoints selectors to
fromApp/toApp (resolved to lineage labels) and fromSG/toSG (resolved to
another group's membership label, a reference the Cilium dataplane resolves
live). Attachments are persisted in a storage-owned CiliumNetworkPolicy
annotation, round-tripped and hidden from the SecurityGroup view like the
marker label. Validation now covers attachments and app/SG peers and rejects
names that collide with reserved Cilium entities.
The membership labels are maintained by a separate controller added in a
follow-up commit; this change covers the API types and the
aggregated-apiserver projection only.
BREAKING CHANGE: SecurityGroup spec.targetRef is replaced by
spec.attachments, and ingress/egress peers use fromApp/fromSG/toApp/toSG
instead of fromEndpoints/toEndpoints. The feature is unreleased, so no
migration is required.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- scope create-github-app-token to the current repo with read-only
contents permission (was org-wide, inheriting the app's blanket
installation permissions); both token steps only read releases/assets
- fail-fast the Talos image upload when the disk is missing
(if-no-files-found: error) so drift fails in build-talos, not later
as a confusing e2e download error
- download the flux install script to a tempfile before executing it,
so a failed curl is not masked by the piped `sudo bash` exiting 0
- fix the regular-PR artifact-source comment (Talos disk comes from
build-talos, pr.patch from finalize) and the TIA scope wording in
docs/agents/e2e-testing.md
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Makes the Keycloak database connection target configurable, so the
connection
can be routed through an external endpoint (for example a connection
proxy such
as PgBouncer or another pgwire proxy) without forking the chart:
- `db.host` / `db.port` — override the host/port otherwise read from the
`keycloak-db-app` secret.
- `db.urlProperties` — sets `KC_DB_URL_PROPERTIES` (previously never
set), e.g.
`?sslmode=verify-full&prepareThreshold=0&sslrootcert=/etc/keycloak-db-ca/ca.crt`.
- `extraVolumes` / `extraVolumeMounts` — mount additional volumes on the
Keycloak container (e.g. a CA for `sslrootcert`).
All new values default to empty, so the rendered StatefulSet is
byte-for-byte
unchanged for existing installs (verified with `helm template`).
Credentials
and database name are still taken from the `keycloak-db-app` secret.
### Release note
```release-note
feat(keycloak): allow overriding the database host/port/URL properties (db.host, db.port, db.urlProperties) and add extraVolumes/extraVolumeMounts, so the DB connection can be routed through an external proxy
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Keycloak database host and port are now configurable through
deployment values
* Added support for custom database connection properties
* Enhanced container storage configuration with additional volume and
mount options
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Implements support for private OCI registry and Git repository
credentials in platform sources. Adds `--platform-source-secret` flag to
the operator and corresponding Helm chart parameter.
Closes#2455
## Changes
### Operator (`cmd/cozystack-operator/main.go`)
- Added `--platform-source-secret` flag for specifying Secret name
- Updated `generateOCIRepository()` to set `spec.secretRef` when secret
is provided
- Updated `generateGitRepository()` to set `spec.secretRef` when secret
is provided
- Secret is expected to be in `cozy-system` namespace
### Helm Chart
- **`packages/core/installer/values.yaml`**: Added
`cozystackOperator.platformSourceSecret` parameter with documentation
- **`packages/core/installer/templates/cozystack-operator.yaml`**: Pass
`--platform-source-secret` flag to operator container
## Usage Example
### For private OCI registry (e.g., GitLab Container Registry):
```bash
# Create registry credentials secret
kubectl create secret docker-registry gitlab-registry-creds \
--namespace cozy-system \
--docker-server=registry.gitlab.example.com \
--docker-username=<username> \
--docker-password=<token>
```
```yaml
# Helm values
cozystackOperator:
platformSourceUrl: 'oci://registry.gitlab.example.com/mygroup/cozystack-packages'
platformSourceRef: 'tag=v1.0.0'
platformSourceSecret: 'gitlab-registry-creds'
```
### For private Git repository:
```bash
# Create Git credentials secret (SSH key or basic auth)
kubectl create secret generic git-creds \
--namespace cozy-system \
--from-literal=username=<username> \
--from-literal=password=<token>
```
```yaml
# Helm values
cozystackOperator:
platformSourceUrl: 'https://github.com/private-org/cozystack-packages'
platformSourceRef: 'branch=main'
platformSourceSecret: 'git-creds'
```
## Testing
Manually verified:
- OCIRepository resource is created with `spec.secretRef` when secret
name is provided
- GitRepository resource is created with `spec.secretRef` when secret
name is provided
- Resources are created without `spec.secretRef` when secret name is
empty (backward compatible)
- Helm chart renders correctly with and without `platformSourceSecret`
parameter
## Checklist
- [x] Code follows project style and conventions
- [x] Operator supports both OCI and Git source types
- [x] Helm chart parameter documented with comments
- [x] Backward compatible (empty secret name is handled correctly)
- [x] Closes referenced issue (#2455)
## Release note
```release-note
feat(operator): add `--platform-source-secret` flag to authenticate the platform package source against private OCI registries (kubernetes.io/dockerconfigjson) and private Git repositories (basic-auth, bearer token, or SSH key). Empty value preserves existing public-registry behaviour.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for configuring a Kubernetes Secret for the platform
source.
* Installer settings and startup options now let you provide a secret
name for Flux source credentials.
* **Bug Fixes**
* Generated platform source resources can now reference the provided
Secret, enabling authenticated access for OCI and Git sources.
* **Documentation**
* Updated configuration docs to explain the new secret setting and
required secret format.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback on #3069:
- ttlSecondsAfterFinished 120 -> 3600 so a failed init Job leaves a 1h
debugging window before the TTL controller collects it (the TTL also
reaps failed Jobs, not just successful ones).
- Load templates/init-script.yaml in the unittest suite: the Job's
checksum/config annotation includes that template, so rendering
init-job.yaml in isolation failed in CI. Select the Job via
documentSelector and assert the 3600 TTL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
Drop the explanatory comment added alongside ttlSecondsAfterFinished;
the rationale lives in the PR/commit history.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The post-install/post-upgrade init Job is a one-shot reconciler of
users/databases/roles. Its hook-delete-policy was `before-hook-creation`
only, which never fires on uninstall, so the completed Job and its Pod
lingered in the tenant namespace forever after the Postgres CR was
deleted (#3055).
Add `ttlSecondsAfterFinished: 120` so the Kubernetes TTL controller GCs
the Job and its Pod ~2 minutes after completion, regardless of the Helm
hook lifecycle. This deliberately avoids touching the hook-timeout
semantics that the bootstrap.recovery path depends on.
Add a helm-unittest suite locking in the TTL and the bootstrap-skip
behaviour to prevent regression.
Fixes#3055
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
