TEMPORARY revert. Restoring persistent EFI/TPM state (4cc60e3) makes
windows.11/2k22/2k25 VMs get a backend-storage PVC that is ReadWriteOnce on the
default StorageClass, which pins the VM to its node and blocks live-migration and
node drains (with evictionStrategy: LiveMigrate this can stall cluster upgrades).
See #3005.
Strip persistent EFI/TPM from the preferences again, but with `yq del` so an
emptied preferredTPM stays `{}` (the v1.8 CRD rejects null) — the bug the original
sed strip caused. Secure Boot and the vTPM stay present, just non-persistent.
Stopgap until vm-state can request an RWX backend (RWX-Block via the upstream
KubeVirt change, or an RWX-Filesystem vmStateStorageClass).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 6ac2601a68)
Merge the AI-generated v1.5.0 changelog with a fuller pass:
- Add an "Upgrade Notes and Required Actions" section (K8s 1.33+, removal
of upgrade.force, GPU permittedHostDevices ownership, MetalLB FRR-K8s /
HTTPS metrics, automatic TLS on externally-published services).
- Expand "Platform Components" with per-component upstream highlights and
CVE references (Flux v2.8 CVE-2026-23990, MetalLB v0.16 CVE-2025-22874),
plus the new frr-k8s subchart and kuberture package.
- Scope the storageClass-immutability entry (#2639) to UI-only enforcement,
with apiserver enforcement tracked in #2657.
- Broaden Development/Testing/CI coverage and add the v1.4.x website docs
backports (#565, #566, #567, #580).
- Correct the loop-device LVM-filter attribution to talm#215 (the prior
ansible-cozystack#51 reference was a dependency bump, not this change).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
The `Release E2E` workflow's build job logged into `iad.ocir.io` using
`secrets.OCI_REGISTRY_USERNAME` / `secrets.OCI_REGISTRY_PASSWORD` —
secret names that don't exist in the repo. `docker/login-action`
therefore received empty credentials and failed at the very first step
with `Username and password required`, before any image was built or any
test ran.
The workflow has **never** succeeded since it was added (2026-05-02):
all three tag runs to date died at this step — `v1.5.0-rc.2`,
`v1.5.0-rc.3`, and `v1.5.0`.
This switches the login to the configured `OCIR_USER` / `OCIR_TOKEN`
secrets — the same ones `build-main.yaml` and `pull-requests.yaml` use
against the same registry — and adds the per-job Docker config isolation
those workflows already use (`Set up Docker config` step +
`DOCKER_CONFIG` env on the login and build steps), so the login doesn't
clobber the shared `~/.docker/config.json` on the self-hosted runner.
### Release note
```release-note
ci(tests): fix OCIR registry login in the Release E2E workflow so it can build and run the full suite on release tags
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **Chores**
* Improved the release workflow with enhanced Docker credential
isolation and configuration handling during the build process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The Release E2E build job referenced secrets OCI_REGISTRY_USERNAME /
OCI_REGISTRY_PASSWORD, which do not exist in the repo, so
docker/login-action received empty credentials and failed every run
with "Username and password required". The workflow has never built
since it was added (2026-05-02): v1.5.0-rc.2, v1.5.0-rc.3 and v1.5.0
all died at this step before any image was built or test run.
Use the configured OCIR_USER / OCIR_TOKEN secrets, matching
build-main.yaml and pull-requests.yaml, and isolate the Docker config
per job (Set up Docker config step + DOCKER_CONFIG on the login and
build steps) so login does not clobber the shared ~/.docker/config.json
on the self-hosted runner.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Implements the tenant Flux sharding design from #2817: a new
`flux-shard-operator` system package that spreads tenant HelmReleases
across multiple helm-controller shards, so a noisy tenant (e.g. a
HelmRelease stuck in infinite remediation) cannot degrade the others.
- **Placement controller** owns the tenant→shard assignment: the unit of
placement is the tenant (all of its HelmReleases share one shard label),
distribution is greedy least-loaded weighted by HelmRelease count, with
pinned tenants, threshold-gated rebalance, paced moves, and never moving
deleting tenants. Watches are metadata-only, so the controller does not
decode the helm-controller status-patch firehose. Assignments are
recorded on tenant namespaces; HelmRelease labels remain the source of
truth on restarts.
- **Shard runtime** is provisioned by the operator: `shardCount`
helm-controller Deployments cloned from the flux-aio `flux` Deployment
and sanitised, so the image and feature-gates stay version-synced
automatically. flux-aio itself is unchanged and keeps owning everything
without a shard key.
- **CREATE-only mutating webhook** (`failurePolicy: Ignore`) stamps the
tenant's shard onto every HelmRelease at admission, so each one is born
on the correct shard regardless of creation path; any miss degrades
gracefully to the placement controller.
- The hand-rolled `flux-tenants` Deployment is retired: the operator
drains it (backfill `tenants` → `shard<i>`) and deletes it once empty;
migration 44 removes a leftover only under the same guard.
Ships with `shardCount: auto` (the default): the operator sizes the
shard count from the tenant HelmRelease count — `K = clamp(ceil(H/100),
1, min(16, T))` with hysteresis (scale up past 120 HR/shard, down under
60) — so small clusters stay at one shard (≈ today's behaviour) and
large fleets shard out automatically. An integer value pins the count
explicitly.
One deviation from the design doc: the webhook/controller scope is the
`sharding.fluxcd.io/key` label rather than
`internal.cozystack.io/tenantmodule` — every tenant HelmRelease is born
with `key=tenants` (stamped by ApplicationDefinition `release.labels`
and chart templates), so the webhook rewrites it and the placement
controller continuously relabels stragglers; `tenantmodule` covers only
the 6 tenant chart modules.
### Screenshots
Not a UI change.
### Release note
```release-note
feat(fluxcd): tenant HelmReleases are now reconciled by operator-managed helm-controller shards; the flux-shard-operator balances tenants across `shardCount` instances and retires the legacy flux-tenants deployment
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced **flux-shard-operator** to shard helm-controller workloads
with deterministic tenant placement, optional rebalance controls, and
autosized shard count.
* Added a **CREATE-only mutating webhook** to stamp the correct shard
label on tenant HelmReleases.
* **Bug Fixes**
* Application updates now preserve existing **flux-shard** label
assignments to prevent unwanted shard re-stamping.
* **Chores**
* Added platform packaging for the new component, including an image
build step.
* Removed the legacy tenants deployment and added a migration to retire
it safely.
* **Tests**
* Expanded unit and chart tests for placement, provisioning, webhook
behavior, and configuration parsing.
* **Documentation**
* Added operator documentation describing configuration and metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `Build cache (main)` workflow (`build-main.yaml`, added in #2948)
failed on
its first run:
https://github.com/cozystack/cozystack/actions/runs/27929277423
The `Set up Buildx` step ran without `DOCKER_CONFIG`, so
`docker/setup-buildx-action` created the docker-container builder under
the
runner's default `~/.docker`. The build step runs with
`DOCKER_CONFIG=$RUNNER_TEMP/.docker` (seeded earlier, holding the OCIR
auth), so
buildx couldn't find the builder and failed immediately:
```
ERROR: no builder "builder-6fb46242-…" found
```
The post-job `docker buildx rm` still succeeded — because the action's
cleanup
runs under the default config where the builder actually lived — which
confirms
the builder existed but in the wrong config store.
This sets `DOCKER_CONFIG` on the `Set up Buildx` step so the builder
instance is
created in the same config the build step reads.
Impact: only the main cache-warmer was broken. PR builds `--cache-from`
this
ref, so a missing cache is a cold build, not a failure — but the
warm-cache
speedup from #2948 wasn't being delivered until this fix.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Improved build pipeline stability by ensuring consistent Docker
configuration across workflow steps. This enhances the reliability of
the container build and deployment process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
During node reprovisioning in tenant Kubernetes clusters, the VMI (and
its hot-plug pod) can outlive the parent VirtualMachine. The upstream
kubevirt-csi-driver treats a missing VM as "detach succeeded", so the
hot-plug pod stays alive and keeps the infra storage device exclusively
attached to the source host — every subsequent attach of that volume
fails, on DRBD with `failed to set source device readwrite`. This was
hit repeatedly in production tenant clusters (see
kubevirt/csi-driver#182).
This PR:
- updates the pinned `kubevirt.io/csi-driver` module to the latest
upstream commit;
- backports kubevirt/csi-driver#184 into the controller wrapper: after
the upstream unpublish returns success, the wrapper re-checks
`VMI.status.volumeStatus` and, if the hot-plug is still reported there
(e.g. because the parent VM is already gone), detaches it from the VMI
directly via the `removevolume` subresource and waits until it
disappears.
The backport is self-contained in the wrapper and can be dropped once
kubevirt/csi-driver#184 merges upstream and the module pin is bumped
past it.
Credit to @mattia-eleuteri for the root-cause analysis and the upstream
fix.
### Screenshots
Not a UI change.
### Release note
```release-note
fix(csi): tenant CSI driver no longer leaves orphan hot-plug volumes exclusively attached to the source host when the VMI outlives its deleted VirtualMachine during node reprovisioning
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced hotplug volume unpublishing with automatic verification to
ensure volumes are fully detached before completion, including retry
logic with up to 2-minute timeout.
* **Dependencies**
* Updated KubeVirt API and CSI driver dependencies to latest versions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The **Auto Patch Release** cron (`.github/workflows/auto-release.yaml`)
walked
*every* `release-X.Y` branch — 21 of them, back to `release-0.9` — and
cut a
`vX.Y.(Z+1)` tag whenever a branch had commits after its latest
published
release. This fires patch releases on long-EOL lines (e.g. `v1.3.6`),
whose
release PRs then sit blocked on unrelated CI failures and accumulate
stray
draft releases, tags and changelog PRs.
This restricts the cron to the **2 newest minor lines**
(`SUPPORTED_LINE_COUNT`).
The window is derived from the live branch list, so it slides
automatically as
new `release-X.Y` branches are cut — once `release-1.5` exists the
window
becomes `{1.5, 1.4}` and the trailing line retires with no further
edits.
Currently active: `release-1.4`, `release-1.3`. Skipped: 19 EOL lines.
### Release note
```release-note
ci(release): the automated patch-release job now only cuts releases for the two newest release branches; older EOL lines no longer receive automatic patch tags.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the automated release workflow to focus patch automation on
only the two newest minor `release-X.Y` maintenance lines.
* Enhanced workflow logging to report the selected branches and any
older/EOL branches skipped.
* **Documentation**
* Refined release documentation to explain the sliding-window selection
for automated patch tagging, including how next patch versions are
determined and when tags are created.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Raises the OpenSSF Scorecard **SAST** check (was **0/10**) by adding
GitHub's recommended CodeQL static analysis for Go. CodeQL is free for
public repositories and is the SAST tool Scorecard recognizes directly.
- Triggers on push to `main`, PRs to `main`, and a weekly schedule (Mon
03:27 UTC, offset from the other Monday cron jobs).
- Results are uploaded to the code-scanning dashboard via
`github/codeql-action/analyze`.
### Why `autobuild`
The Go analysis uses **`build-mode: autobuild`**. Go's CodeQL extractor
only supports `autobuild` or `manual` (not `none`), so autobuild is the
low-friction choice. It discovers every `go.mod` in the tree and
extracts each module separately — the root module, `api/apps/v1alpha1`,
and the three nested image modules a root `go build ./...` never reaches
— giving broader coverage than a single root build. Before falling back
to the Go build it first attempts `make` from the repo root, which runs
the default Makefile goal (including a network `git fetch upstream
--tags`). `setup-go` pins the toolchain to the `go.mod` version so the
extractor uses the project's Go rather than whatever the runner ships.
### Security posture
- Top-level token is **read-only**; the `analyze` job opts into
`security-events: write` (plus `contents: read`, `actions: read`) only.
- `actions/checkout` is pinned to **v6.0.2** (matching `scorecard.yml`)
with `persist-credentials: false`.
- `codeql-action/init` and `/analyze` are pinned to the **v4.36.2 SHA**
— the same commit as `scorecard.yml`'s `upload-sarif`, keeping every
codeql-action ref in the repo on one commit and clear of the v3
deprecation warning.
### Verification
- `actionlint`: **exit 0**.
- Valid YAML.
- No `run:` steps → no untrusted-input injection surface.
> SAST is a ramping check: Scorecard credits the CodeQL workflow's
presence immediately and the score climbs toward full as subsequent
merged PRs get analyzed.
### Release note
```release-note
ci: add CodeQL static analysis (SAST) for Go, running on push/PR to main and weekly
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Enabled automated security scanning via CodeQL on pushes, pull
requests targeting the main branch, and on a weekly schedule.
* Added workflow controls for concurrency and longer execution time,
with support for Go-based analysis using an autobuild setup.
* Updated the SARIF upload step to use the latest CodeQL action version
to improve reliability while maintaining secure, minimal permissions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Promotes universal project guidance that previously lived only in a
local
(gitignored) `CLAUDE.md` into the committed agent docs, so every
contributor and
every tool (Codex/Cursor/etc., not just Claude) gets it from a fresh
clone.
- `overview.md`: Build & Development Commands (root + per-package make
targets,
build env vars), Values schema generation, a Testing section, Domain
references
- `contributing.md`: a "Verify Before Considering Done" matrix + Review
Expectations
- Dedup: `overview.md`'s Git Commits block now points to
`contributing.md`
> Note: `overview.md` links to `docs/agents/e2e-testing.md` added in
#2932 — merge
> that one first or together.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Expanded contribution guidance with a “Verify Before Considering Done”
pre-submission checklist and clearer expectations for including
behavioral regression tests in bug-fix PRs.
* Broadened build/development documentation with a “Build and
Development Commands” guide, including required environment variables,
values-schema generation, and test entry points.
* Updated contribution workflow notes by consolidating commit-convention
guidance and adding pointers to task-specific design references.
* Added a Claude entry-point document that directs to the main agents
guide and explains local override placement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds `docs/agents/e2e-testing.md` — a routed agent doc capturing the
e2e/CI
stabilization conventions built up over the last few months:
- No retries on deterministic CI steps (only pure-infra `Prepare
environment` retries)
- Event-driven backstops instead of fixed `sleep`; no `EXIT`/`RETURN`
traps
- Fail-fast HelmRelease-readiness gating + the `status.history`
remediation guard
- First-class failure diagnostics and a reviewer checklist
- Test-Impact Analysis (TIA) and its real scope: it trims only the
per-app loop,
runs after install, and most PRs legitimately escalate to the full suite
Wired into `AGENTS.md` so every agent reads it before e2e/CI work.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Introduced comprehensive E2E testing conventions document with
standards for writing, modifying, and reviewing end-to-end tests and CI
processes.
* Documented best practices for test reliability, including guidelines
on retry behavior, event-driven test validation, and cleanup discipline.
* Updated process documentation to reference new E2E testing
conventions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Why
Follow-up to #2509. PR #2509 exposed five HelmRelease generation knobs
as `cozystack-operator` flags
(`--helmrelease-{interval,retry-interval,install-timeout,upgrade-timeout,max-history}`).
The other HelmRelease-generating path — `cozystack-api`'s
`convertApplicationToHelmRelease` in
`pkg/registry/apps/application/rest.go` — was deliberately left out of
#2509's scope and was flagged in review:
> `pkg/registry/apps/application/rest.go` (the other
HelmRelease-generating path) still hardcodes `Interval: 5m`,
`Remediation{Retries:-1}`, no `Strategy`. Out of this PR's scope; worth
a Followups bullet so the rationale doesn't drift across paths.
So today the api-side path hardcodes:
- `Spec.Interval = 5m`
- `Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`
- no `Strategy`
- no `MaxHistory`
Only `HelmInstallTimeout` (per-Application annotation override) flows
through `ReleaseConfig`. This PR brings the api-side to feature parity
with the operator: same flag names, same defaults, same validation, same
generated HelmRelease shape.
## What
### New cozystack-api flags
Names, types, defaults, and validation match the operator's flags
introduced in #2509 verbatim, so both binaries can be tuned with the
same vocabulary.
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
Validated in `CozyServerOptions.Complete()` via
`config.ParsePositiveDuration` so a misconfigured server restarts loudly
instead of waiting until the first Application is created. `MaxHistory <
0` is rejected; `0` is honored (unlimited per Helm semantics, matching
the operator).
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (with `RetryInterval` from the new flag) instead of
`Install/Upgrade.Remediation{Retries: -1}`. Functionally equivalent for
this path (which never relied on remediation firing), but **decouples
failed-install retry timing from `Spec.Interval`** the same way the
operator change did.
`Install.Remediation` / `Upgrade.Remediation` stay nil: retries run
through the single `Strategy.RetryInterval` path, matching
cozystack-operator's `PackageReconciler`. Reintroducing
`Remediation{Retries:-1}` "for safety" would add a second, conflicting
retry mechanism and break parity with the operator path. A regression
test guards this. (The deployed helm-controller CRD has no rule
forbidding a `Strategy` alongside a `Remediation` entry — this is a
parity/clarity guard, not a CRD-validation one.)
### Per-Application timeout override is preserved (and extended)
The `release.cozystack.io/helm-install-timeout` annotation on
`ApplicationDefinition` (`HelmInstallTimeout` in `ReleaseConfig`) still
wins over the new global `HelmReleaseInstallTimeout` /
`HelmReleaseUpgradeTimeout` defaults when set, applying to **both**
`Install.Timeout` and `Upgrade.Timeout`. kubernetes-rd and tenant-rd
carry it today (Kamaji bootstrap; seaweedfs-db CNPG bootstrap).
A new `release.cozystack.io/helm-upgrade-timeout` annotation overrides
**only** `Upgrade.Timeout`, winning over the value the install
annotation would otherwise apply — so a kind can carry an asymmetric
install/upgrade budget. No `ApplicationDefinition` sets it today, so
behavior is unchanged for every existing kind.
`Install.Timeout` and `Upgrade.Timeout` are now always populated (from a
per-app override or the global default), matching the operator's
`buildHelmReleaseSpec`. The previous "stay nil when zero" branch was
removed — it was a test shim that kept empty-`ReleaseConfig{}` fixtures
green at the cost of api/operator drift.
### Shared validator
`parsePositiveDuration` is lifted from `cmd/cozystack-operator/main.go`
into `pkg/config` (exported as `ParsePositiveDuration`). Both
`cozystack-operator` and `cozystack-api` mains now share one validator
with identical reject rules (zero / negative / malformed / empty). The
table-driven test moves with it. The annotation parser is shared the
same way: `ParseHelmTimeoutAnnotation` validates both the install- and
upgrade-timeout annotation values against Flux's unit rules.
## Production safety
- Static flag defaults match #2509 exactly. **With stock defaults, every
api-generated HR changes shape**: it gains
`Strategy.Name=RetryOnFailure` + `RetryInterval`, an explicit
`MaxHistory`, and an explicit `Install/Upgrade.Timeout`. This is the
intended parity with the operator path, not a no-op.
- **Timeout-default change**: because `Install/Upgrade.Timeout` is now
always set, the effective install/upgrade timeout for api-generated HRs
grows from the implicit flux default (5m, field was previously nil) to
the explicit `--helmrelease-{install,upgrade}-timeout` default of `10m`.
This is desired — it matches the operator path.
- Same intentional retry-cadence change as #2509: failed-release retry
cadence on api-generated HRs drops from ~5m (previously coupled to
`Spec.Interval`) to 30s (`RetryInterval` default). Healthy-release
reconcile cadence is unchanged at 5m.
- Per-Application `HelmInstallTimeout` override semantics intact —
kubernetes-rd's long Kamaji bootstrap path continues to work;
tenant-rd's seaweedfs-db bootstrap budget is unchanged.
## Verification
- `go build ./...` clean
- `go vet ./...` clean
- `go test ./pkg/config/... ./pkg/cmd/server/...
./pkg/registry/apps/application/... ./cmd/cozystack-operator/...
./internal/operator/...` — all pass
- New / updated unit tests mirror the operator's spec tests:
- `TestConvertApplicationToHelmRelease_BuildsSpecFromConfig` — full HR
shape from config (mirrors `TestBuildHelmReleaseSpec`)
- `TestConvertApplicationToHelmRelease_ZeroMaxHistory` — `MaxHistory=0`
survives as pointer-to-0 (mirrors
`TestBuildHelmReleaseSpecZeroMaxHistory`)
- `TestConvertApplicationToHelmRelease_PerAppTimeoutOverridesGlobal` —
install-annotation override wins over globals (symmetric + asymmetric)
- `TestConvertApplicationToHelmRelease_UpgradeTimeoutAnnotation` —
upgrade-only override, asymmetric precedence over the install
annotation, and the install-sets-both regression guard
- `TestConvertApplicationToHelmRelease_AppliesReleaseConfigTimeout` —
`Remediation.Retries == -1` assertions replaced with `Strategy.Name ==
RetryOnFailure` + `Remediation == nil`
- `TestParsePositiveDuration` / `TestParseHelmTimeoutAnnotation` — flag
and annotation validation, including which error branch fires
- Pre-existing test failures in `pkg/apiserver` (randfill panic) and
`pkg/lineage` reproduce on `main` without these changes — not introduced
here.
## Test plan
- [ ] CI E2E install passes with default values (no flags set on
cozystack-api Deployment)
- [ ] HR objects generated from Application resources in a healthy
cluster render with `Strategy.Name=RetryOnFailure` and
`RetryInterval=30s`
- [ ] kubernetes-rd Application's HR still gets
`Install/Upgrade.Timeout=15m` (or whatever its annotation says) —
overrides the global default
## Followups (deliberately not bundled)
- **Helm chart exposure** — equivalent of #2509's
`packages/core/installer/values.yaml` knobs, but for the api server.
Touches a different chart (`packages/core/cozystack-api` or wherever the
api Deployment lives). Separate PR keeps the diff focused on the Go
side.
- **Per-Application override of the remaining globals** — interval /
retry-interval / max-history have no annotation hook yet; adding them
would require new annotations and is out of scope until a real need
surfaces.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added server-wide HelmRelease timing flags and defaults (interval,
retry interval, install/upgrade timeouts, max history), with
per-application timeout overrides.
* HelmRelease generation now consistently applies a retry-on-failure
strategy with explicit install/upgrade timeouts.
* **Bug Fixes**
* Stricter validation for duration-related flags to reject malformed,
zero, or negative values.
* **Tests**
* Expanded unit tests for flag parsing/validation, HelmRelease spec
generation (including max-history=0 handling), timeout override
precedence, and invalid Git ref handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The "Set up Buildx" step in build-main.yaml ran without DOCKER_CONFIG, so
docker/setup-buildx-action created the docker-container builder under the
default ~/.docker. The build step runs with DOCKER_CONFIG=$RUNNER_TEMP/.docker
(seeded before the builder existed and holding the OCIR auth), so it could not
find the builder and failed immediately with `no builder "builder-<uuid>"
found`. The post-job `docker buildx rm` still succeeded because the action's
cleanup runs under the default config where the builder actually lived.
Set DOCKER_CONFIG on the Set up Buildx step so the builder instance is created
in the same config the build step reads.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The shardCount description and the shard.resources value overflowed their columns, pushing the internal separators out of line. Reflow every column to its widest cell so the raw table renders cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
BuildShardDeployment clones the flux-aio pod spec, which carries a required podAntiAffinity on app.kubernetes.io/name=flux. Inherited onto a shard pod it forbids scheduling on the flux-aio node, so on a single schedulable node every shard stays Pending and the tenant plane stalls (no ready shard -> assignments deferred -> HelmReleases stuck on key=tenants).
Drop the podAntiAffinity during sanitisation while keeping any benign nodeAffinity (e.g. os=linux). The fixture now carries flux-aio's affinity and TestBuildShardDeployment asserts the strip.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
release.cozystack.io/helm-install-timeout sets both Install.Timeout and
Upgrade.Timeout to one value. Add release.cozystack.io/helm-upgrade-timeout
so an ApplicationDefinition can override only Upgrade.Timeout — letting a
kind keep a short install budget but a longer upgrade budget (or vice
versa). When set it wins over the value the install annotation would
otherwise apply to the upgrade side.
- config: add HelmUpgradeTimeoutAnnotation + ReleaseConfig.HelmUpgradeTimeout;
generalize ParseHelmInstallTimeoutAnnotation to ParseHelmTimeoutAnnotation
since it now parses both annotations (identical Flux unit validation).
- server: parse the new annotation at startup with the same loud-fail
behavior; update --helmrelease-upgrade-timeout help.
- rest: apply HelmUpgradeTimeout after the install-annotation copy so the
upgrade-specific value wins; restore the accurate carrier comment.
- tests: cover the upgrade-only override, the asymmetric precedence, and
the install-sets-both regression guard.
Behavior is unchanged for every existing ApplicationDefinition (none set
the new annotation today).
Address review feedback from coderabbitai on rest.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The ParsePositiveDuration table exercised all error inputs but did not
distinguish which branch fired. Add an errMatch field so the table
pins "must be > 0" for zero/negative and "invalid duration" for
malformed/empty inputs, mirroring TestParseHelmInstallTimeoutAnnotation
and keeping the two parsers' diagnostics from drifting.
Address review feedback from coderabbitai on config_test.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
HelmReleaseMaxHistory carried omitempty, so a value of 0 (which Helm
treats as "unlimited") would be dropped on marshal and become
indistinguishable from an unset field. The runtime path (start.go
always populates it from --helmrelease-max-history) is safe today, but
drop omitempty so the explicit "unlimited" intent survives if
ReleaseConfig is ever serialized.
Address review feedback from coderabbitai on config.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
gofmt -w on the two touched test files: the case-table struct fields in
rest_helmrelease_spec_test.go and main_test.go were misaligned, and
main_test.go carried a trailing blank line. CI runs no gofmt gate
(pre-commit only runs make generate), so this would otherwise land
unformatted.
Address review feedback from lexfrei.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The convertApplicationToHelmRelease comment listed only kubernetes-rd
as carrying release.cozystack.io/helm-install-timeout, dropping
tenant-rd. tenant-rd/cozyrds/tenant.yaml sets the 15m annotation for
the seaweedfs-db CNPG bootstrap whose first reconcile exceeds the flux
default. Restore the accurate two-carrier list and rationale.
Address review feedback from lexfrei on rest.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The XValidation justification was factually wrong: the deployed
helm-controller CRD has no rule rejecting a Strategy alongside a
Remediation entry (only .retryInterval is gated on Strategy.Name), so
the claimed "silently break every HR" safety net does not exist.
Reword the comment in all three test files to the real reason
Remediation stays nil: retries are driven by a single
Strategy.RetryInterval path, matching cozystack-operator's
PackageReconciler.buildHelmReleaseSpec.
Address review feedback from lexfrei on rest_helmrelease_spec_test.go,
rest_timeout_test.go and internal/operator/package_reconciler_test.go.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Production change:
- convertApplicationToHelmRelease now sets Install/Upgrade.Timeout
unconditionally (matching cozystack-operator's buildHelmReleaseSpec).
The previous "stay nil if zero" branch was a leaky test shim — it kept
fixtures with empty ReleaseConfig{} green at the cost of api/operator
drift.
Test changes:
- newRESTForTimeout seeds the new HelmRelease* globals with
production-shaped defaults so the unset-annotation case exercises
"global default applies" rather than "field stays nil".
- TestConvertApplicationToHelmRelease_PerAppTimeoutOverridesGlobal is
table-driven and now covers asymmetric globals (install != upgrade)
with and without per-app override — guards against a regression that
only assigns one side under override.
- New start_test.go: table-driven validator coverage for each
malformed/non-positive duration flag, the negative MaxHistory branch,
the MaxHistory=0 unlimited branch, asymmetric timeout acceptance, and
a smoke test that NewCozyServerOptions defaults validate.
Refactor:
- Extracted parseAndValidateHelmReleaseFlags from Complete() so the
flag-validation portion is unit-testable without kubernetes I/O.
Documentation:
- Reworded --helmrelease-install/upgrade-timeout help so admins reading
--help see the dual-purpose annotation explicitly (one annotation
overrides both timeouts together).
- Dropped PR-#2509 references and "preserving the pre-#2509 behaviour
for tests" comments — kept the technical rationale (Strategy.Name
vs Remediation.Retries:-1 + 5m Interval coupling) on its own merits.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
convertApplicationToHelmRelease now builds the HelmRelease spec from
ReleaseConfig instead of hardcoding values, bringing it in lockstep
with cozystack-operator's PackageReconciler.buildHelmReleaseSpec
(PR #2509). The drift between the two HelmRelease-generating paths
was the rationale for this follow-up.
Changes to the generated spec:
- Spec.Interval reads ReleaseConfig.HelmReleaseInterval (was
hardcoded 5m).
- Spec.MaxHistory set as &maxHistory pointer from
ReleaseConfig.HelmReleaseMaxHistory; 0 survives as pointer-to-0
(unlimited per Helm semantics) - mirrors the operator's
TestBuildHelmReleaseSpecZeroMaxHistory.
- Install.Strategy{Name: RetryOnFailure, RetryInterval} replaces
Install.Remediation{Retries: -1}. Same for Upgrade.Strategy.
Functionally equivalent to "retry forever, never remediate" for
the api-side path (which also never relied on remediation
firing) but decouples failed-install retry timing from
Spec.Interval.
- Install.Remediation / Upgrade.Remediation stay nil:
helm-controller's XValidation rejects RetryOnFailure +
RetryInterval alongside a Remediation entry, so reintroducing
Retries:-1 "for safety" would silently break every HR.
Install/Upgrade.Timeout precedence:
- per-Application HelmInstallTimeout (annotation-driven) wins
when non-zero (preserves the kubernetes-rd opt-in path);
- otherwise HelmReleaseInstallTimeout / HelmReleaseUpgradeTimeout
globals apply;
- when both are zero, the field stays nil and flux defaults
apply (preserves the legacy code path for tests that
hand-construct ReleaseConfig without populating the new globals).
Tests:
- rest_timeout_test.go: Remediation.Retries == -1 assertions
replaced with Strategy.Name == RetryOnFailure +
Remediation == nil, matching the operator's spec test.
- rest_helmrelease_spec_test.go: new file mirroring
TestBuildHelmReleaseSpec / TestBuildHelmReleaseSpecZeroMaxHistory
from internal/operator/package_reconciler_test.go, plus a third
case asserting per-Application HelmInstallTimeout overrides the
global Install/Upgrade timeout defaults.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Wire five flags into cozystack-api that mirror the cozystack-operator
flags introduced in PR #2509:
--helmrelease-interval (default 5m)
--helmrelease-retry-interval (default 30s)
--helmrelease-install-timeout (default 10m)
--helmrelease-upgrade-timeout (default 10m)
--helmrelease-max-history (default 5)
CozyServerOptions.Complete() validates each duration via
config.ParsePositiveDuration so a misconfigured operator restarts
loudly instead of waiting until the first Application is created,
and rejects MaxHistory < 0 (0 is valid: unlimited per Helm
semantics, matching the operator).
Validated values are written into every ReleaseConfig built from
the ApplicationDefinitionList. The per-Application
HelmInstallTimeout (annotation-driven) still wins over the new
global Install/Upgrade timeout defaults; this commit only plumbs
the values - the consumer in convertApplicationToHelmRelease lands
in the next commit.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Lift parsePositiveDuration from cmd/cozystack-operator/main.go into
pkg/config so cozystack-api can reuse the same startup validator
without duplication. Operator main.go now imports and delegates;
the table-driven test moves to pkg/config/config_test.go.
Extend ReleaseConfig with the five HelmRelease generation knobs
the cozystack-operator already exposes via flags (PR #2509):
HelmReleaseInterval, HelmReleaseRetryInterval,
HelmReleaseInstallTimeout, HelmReleaseUpgradeTimeout,
HelmReleaseMaxHistory. The existing per-Application
HelmInstallTimeout (set from the
release.cozystack.io/helm-install-timeout annotation) keeps its
override semantics - it still wins over the new global
Install/Upgrade timeout defaults when set.
No behaviour change in this commit; the new fields are wired up in
follow-up commits.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
Consolidated landing branch for a set of interdependent platform/CI
stabilization fixes. It began as the source PRs in the table below (in
dependency order) and has since grown with review-driven fixes and a few
production-behavior changes — the full current inventory is under
**Scope** below. Landing them as one unit lets CI run against the real
combined state instead of a fragile bottom-up merge train.
## Source PRs (dependency order)
| # | Commit | Source PR | Addresses |
|---|--------|-----------|-----------|
| 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi
crashloop during cert provisioning |
| 2 | test(e2e): bucket.bats port-forward + S3 client reliability |
#2944 | flaky bucket test |
| 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop
volumeSizeLimitMB | #2943 | missing bucket/access classes |
| 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps)
| #2874 | cilium "IP already in use" leak |
| 5 | refactor(build): mode=max registry cache + main-only warmer |
#2938 | build cache |
| 6 | ci(build): isolate each PR build on its own ephemeral runner VM |
#2939 | shared-buildkit contention |
| 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline |
#2928 | LINSTOR provisioning timeouts |
| 8 | test(metallb): assert digest-pinned image form, not version
literal | #2873 | brittle metallb assertion |
## Why consolidated
These form a dependency DAG (verified from CI logs): e.g. 2943 needs
2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested
bottom-up, each lower PR runs with none of the fixes above it and can't
go green alone. This branch carries all of them, so CI runs against the
real combined state.
## Verification status
CI is green on the latest head (`5653c30`): full-suite E2E passes
end-to-end and is reproducible (2 of 3 runs on this SHA green). The one
red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the
sandbox, unrelated to the diff — a different environmental subsystem
failed each run (details in the comments below). The earlier 3-hour
crust-gather snapshot hang is fixed in `5653c30`.
## Scope beyond the original 8
The branch has grown past the 8 source PRs above with review-driven
fixes and a few production-behavior changes. For reviewer transparency,
the full set:
**Review fixes (@lexfrei review):**
- **B1 (blocker)** — `8f41910` converts the SeaweedFS
`-lock`/`-readonly` BucketClasses and the s3 service-name override into
`patches/`, wired into `make update`, so a re-vendor no longer drops
them.
- **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods
(both delete remedies were RBAC-forbidden).
- **FU3** — `014de3f` adds a unit test for the HelmRelease update
conflict-retry path.
- **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any
non-terminal owner phase, not just `Running`.
- **FU7** — `f3914d2` sets `persist-credentials: false` on the
build-cache checkout (SHA-pinning is handled repo-wide in #2849).
**Production-behavior changes (not in the table above):**
- VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring
(`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops
install-time eviction churn.
- cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease
update path (`728f676`).
**E2E robustness follow-ups:** per-test crust-gather snapshots
(`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node
single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget
(`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin
(`95f42aa`).
**Not addressed (by decision):** FU2 — the s3 Service keeps
`seaweedfs.name`-based naming via patch (rendered output unchanged;
cozystack renders `fullname == seaweedfs`, so it matches the
`componentName`/`fullname` siblings in practice). FU6 — fork PR build
push is handled by the existing mirror-to-same-repo-branch workflow.
## Relates to
#2946#2944#2943#2874#2938#2939#2928#2873 — if this lands, those
can be closed; otherwise they remain the granular per-PR review path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added CI build cache warming for main branch builds.
* Added a best-effort in-cluster Cilium endpoint self-heal watchdog for
e2e installs.
* Extended SeaweedFS COSI with object-lock support plus separate
readonly access.
* **Improvements**
* Reduced e2e install flakiness with a shared readiness deadline, safer
waits, and improved cleanup/diagnostics capture.
* Prevented install-time churn by switching VPA update mode to
**Initial**.
* Added container startup probes and updated SeaweedFS volume sizing.
* **Tests**
* Improved e2e robustness (etcd backup schedule waiting, digest-pinned
image checks, and corrected S3 port-forwarding).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an operator-facing way to serve platform services and the root
tenant's ingress under a pre-existing wildcard TLS certificate instead
of minting per-host ACME certificates. Implements the "native
references" design decided on #2812.
A single value, `publishing.certificates.wildcardSecretName`, names a
TLS Secret the operator has already created in the publishing namespace
(`tenant-root` by default). Only the name travels through the
`cozystack-values` channel — the certificate and private key are never
broadcast, unlike the public `kube-root-ca` that already rides it.
When set:
- **ingress-nginx path** (`gateway.enabled=false`): the publishing
tenant's ingress controller serves the Secret as its
`--default-ssl-certificate`, and the platform service Ingresses drop
their cert-manager cluster-issuer annotation and `tls` block. No
per-host ACME.
- **Gateway API path** (`gateway.enabled=true`): a new `TenantGateway`
cert mode, `existingSecret`, references the Secret in the wildcard/apex
listener `certificateRefs` and mints no Issuer or Certificate. Switching
into this mode garbage-collects any ACME machinery left from
`http01`/`dns01`.
Scope is the root tenant — everything is same-namespace, so no RBAC or
NetworkPolicy changes. Extending wildcard mode to child tenants needs
the Secret replicated into each tenant namespace and is left as a
follow-up.
Closes#2812. Part of #2811.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(platform): operators can serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate via `publishing.certificates.wildcardSecretName`, instead of minting per-host ACME certificates
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an "existingSecret" TLS certificate mode and a wildcardSecretRef
option for TenantGateway and platform values to use a pre-existing
wildcard TLS Secret.
* **Platform / Templates**
* Introduced operator "wildcard secret" mode across Gateway and many
Helm charts (Ingress, apps) to skip per-host ACME provisioning and
optionally wire a default SSL cert.
* **Tests**
* Added extensive chart and controller tests covering existingSecret
behavior and cert-mode transitions.
* **Documentation**
* Updated gateway docs describing the new wildcard secret mode and
requirements.
* **Chores**
* Added Helm unit test makefile targets to several packages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback from lexfrei on .github/workflows/auto-release.yaml:82: the skipped-branches log used a lexicographic sort (release-0.9 after release-0.41). Reuse the existing lineNum helper so the log lists EOL branches in numeric order.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback from lexfrei on docs/release.md: the Patch Releases section described only the manual process, so add an "Automated Patch Releases" subsection covering the nightly cron, the new-commits trigger, the 2-newest-lines support window, and the manual tag-push escape hatch for EOL lines.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Auto Patch Release cron walked every release-X.Y branch (21 of them,
back to release-0.9) and cut a vX.Y.(Z+1) tag whenever a branch had
commits after its latest published release. That fired patch releases on
long-EOL lines (e.g. v1.3.6), whose release PRs then sit blocked on
unrelated CI failures and accumulate stray draft releases, tags and
changelog PRs.
Restrict the cron to the SUPPORTED_LINE_COUNT (=2) newest minor lines.
The window is derived from the branch list, so it slides automatically as
new release-X.Y branches are cut — once release-1.5 exists the window
becomes {1.5, 1.4} and the trailing line retires with no further edits.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh)
ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect
can hang indefinitely — observed wedging the E2E install step for ~2h35m after a
test failure until the job-level cancel, with no output after the capture line.
`|| true` only catches a bad exit code, never a hang.
Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s
later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still
beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The build-main cache warmer is a privileged workflow (packages:write + an OCIR
push token) and never pushes git, so set persist-credentials: false to keep the
checkout's GITHUB_TOKEN out of .git/config where a later build step could read
it. (SHA-pinning the actions is handled repo-wide in #2849.)
Addresses non-blocking follow-up 7 on #2948.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
REST.Update wraps the HelmRelease write in retry.RetryOnConflict so a stale
resourceVersion (the helm-controller racing the status subresource) is
refreshed and retried instead of surfacing a 409. Add a unit test using a fake
client whose interceptor returns one Conflict on the first Update and succeeds
on the second, asserting Update returns nil and writes exactly twice.
Addresses non-blocking follow-up 3 on #2948.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The leak-healer only refused to disconnect an endpoint whose owning pod was
Running, so a Pending owner that already held a live IPAM allocation
(status.podIP is populated once the sandbox is created, before the pod goes
Running) could have its endpoint evicted mid-startup. Refuse for any
non-terminal phase (!= Succeeded && != Failed); only a terminal owner is
safely past needing its endpoint.
Addresses non-blocking follow-up 4 on #2948.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
No code change. The 'debug' PR label was pinning E2E to the contended
self-hosted runner; with it removed this synchronize event re-evaluates
runs-on to the ephemeral oracle-vm pool. Empty commit because the workflow
only triggers on opened/synchronize/reopened (not label changes) and a plain
re-run reuses the original event payload (stale labels).
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.
Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.
Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).
Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The healer's reserved/infra-endpoint remedy and its disconnect-failure
fallback both run `kubectl delete pod` to reschedule the wedged pod onto a
free IP, but the ClusterRole granted pods only get/list — so the delete was
forbidden and the pod stayed wedged. Add delete to the pods rule.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The restored COSI -lock/-readonly BucketClass/BucketAccessClass and the s3
Service name override lived only as raw edits in the vendored chart tree, so
the next `make update` (rm -rf charts && curl | tar && patch) would silently
drop them again — the same way the 4.31 bump did. Capture both as patch files
under patches/ and wire them into the Makefile update target, matching the
existing s3-tls-main-port / cosi-provisioner-sa-name customizations. No
rendered output changes; this only makes the customizations survive re-vendor.
Addresses the B1 review blocker on #2948.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
crust-gather can only reach a tenant Kamaji cluster through the test's
localhost port-forward (it connects directly to the kubeconfig server URL —
there is no apiserver-proxy mode — and the in-cluster URL is unreachable from
the runner). cozytest.sh's generic EXIT trap runs in the parent shell AFTER
the failing test subshell has exited and its port-forward is reaped, so a
collect there hits a dead socket and crust-gather bails before writing
anything (every tenant snapshot so far came back empty).
Capture the tenant from an EXIT trap registered INSIDE run_kubernetes_test
instead: it fires during THAT subshell's exit, while the supervised
port-forward is still alive. Verified end-to-end against the real
hack/cozytest.sh under dash (CI's /bin/sh) in an ubuntu:22.04 container —
the in-subshell trap fires on failure with the PF alive, the tenant snapshot
lands in the same snapshots/<test>/ dir as the host snapshot, the generic
cozy_cleanup still runs, and there is no pipeline deadlock.
Drop the now-dead tenant loop from cozytest.sh's generic trap (it could only
ever see a reaped PF); it keeps the host snapshot.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
A single `kubectl port-forward` to the tenant Kamaji API silently dies on
an SPDY/connection drop and never reconnects. The tenant apiserver itself
stays healthy (control-plane pods observed at 0 restarts while localhost
refused connections), so this is a client-side port-forward death, not a
control-plane outage. Once it dies every later tenant kubectl op wedges:
a long `kubectl wait` burns its full timeout against the dead socket
(observed: nfs-test-pod hit its 10m deadline) and the failure-path
`describe`/`get events` print 'connection refused', destroying the
diagnostics too.
Run the port-forward under a respawn loop so a drop self-heals within ~1s;
`kubectl wait` retries across the gap and the in-test failure dumps work
again. Track the supervisor PID in a pidfile so cozy_cleanup() can stop it
on a failed run (the success path kills it directly), and reap the child
port-forward with pkill in both paths.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The aggregated API server translates app CR updates (Etcd, Postgres, ...)
into a read-convert-update of the backing HelmRelease. Flux's
helm-controller continuously writes the HelmRelease status, which shares
the object's resourceVersion, so a CR update that lands while a prior
reconcile is still in flight reads a resourceVersion that goes stale
before the Update and the request fails with a 409 Conflict
("the object has been modified; please apply your changes to the latest
version and try again").
The HelmRelease spec is fully derived from the Application the caller
just applied, so a stale-resourceVersion conflict is never a real spec
conflict here. Wrap the Update in retry.RetryOnConflict, refreshing the
live resourceVersion on each conflict before retrying. The no-conflict
path is unchanged, and this mirrors the existing RetryOnConflict usage in
workloadmonitor_controller.go.
Surfaced by the etcd.bats e2e test, which applies the same singleton
Etcd resource three times back-to-back and intermittently failed the
third apply with the 409.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 76410599de)
Address non-blocking review feedback from lexfrei: bump actions/checkout to the v6.0.2 SHA already used by scorecard.yml so the security workflows track one version, and remove the hardcoded 'codeql-' prefix from the concurrency group (github.workflow already expands to 'CodeQL').
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Andrei Kvapil