Every release-1.5 PR Build job dies at the 30-minute timeout, and all of
it is spent in one target. `image-ubuntu-container-disk` builds a disk
per Kubernetes minor -- six of them -- and each spends ~4-5min inside a
libguestfs appliance installing that minor's kubelet/kubeadm into its own
copy of the cloud image. Serially that is ~28min of the 30min budget:
v1.35 6:00, v1.34 4:21, v1.33 4:27, v1.32 4:39, v1.31 4:42, v1.30 cancelled
None of it is cached, and no warmer will fix that on its own. Only
build-main.yaml sets WRITE_CACHE=1, and main dropped this image from
`make image` in 39cdc8866 six days before release-1.5 was cut, so the
:*-buildcache refs it reads have never been written -- the CI log shows
all six importers failing with "not found". release-1.5 is now the only
branch that builds this image at all.
The six builds share no per-version work, so run them concurrently
instead. A sub-make carries its own -j, keeping the root `make build`
serial, and buildkit dedups the shared guestfish and cloud-image stages
across the concurrent solves. --output-sync=target groups each version's
log under its target name; six interleaved buildx progress streams have
no per-solve identifier and are unreadable. Verified the per-version
buildx/echo/rm command set is byte-identical to the loop it replaces
across PUBLISH_VERSIONED, PUBLISH_FLOATING and WRITE_CACHE combinations.
Moving off the 4cpu/16gb shape because six concurrent libguestfs
appliances want ~9gb of RAM and a core apiece. The comment justifying
the small shape assumed a warm cache, which no longer holds here.
This also stops swallowing failures. The loop separated versions with
`;`, so make saw only the last version's exit status: a failed build for
any other version left its committed .tag file stale and the job green.
Under the sub-make a failure in any version fails the target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The flux-instance template rendered spec.distribution.artifact
unconditionally, unlike the sibling optional fields (artifactPullSecret,
imagePullSecret, variant). The umbrella default is empty to use the
operator's embedded manifests (the air-gapped default from #964), but an
empty value renders as YAML null. In the flux-operator v0.50.0 CRD the
field is optional yet typed 'string' with pattern ^oci://.*$ and is not
nullable, so the rendered null (like an empty string) fails schema
validation and the API rejects the FluxInstance.
Guest Kubernetes clusters with the fluxcd addon enabled could therefore
never install it: the sub-release looped install/uninstall, the parent
kubernetes release stayed InProgress, and the cluster never reached Ready.
Guard the field so an empty value is omitted (the operator falls back to
its embedded manifests, preserving the air-gapped default) while a
non-empty artifact still renders. The template lives in the vendored
subchart, so the change is carried as a patch applied by 'make update'.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit da47b70e2c)
Two related issues caused the dashboard SPA to break with CORS errors
after the kc-access cookie expired:
1. KeycloakClient for dashboard had no webOrigins set, so the operator
never touched the field in Keycloak after initial creation. After
any rootUrl change the existing webOrigins value became stale and
blocked CORS from the SPA. Setting "+" tells Keycloak to derive
allowed origins from redirectUris.
2. oauth2-proxy returned 302 -> Keycloak /auth for unauthenticated
XHR/fetch on /api and /apis. The login page does not emit
Access-Control-Allow-Origin, so the SPA's fetch failed with CORS.
--api-route makes oauth2-proxy answer 401 without redirect, so the
SPA can handle the re-auth itself via top-level navigation.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
(cherry picked from commit be8f6b4739)
A leftover debug field "foo" in the <release>-urls Secret template
violates the core/v1 Secret schema. It went unnoticed under client-side
apply but breaks Helm upgrades under server-side apply with
".foo: field not declared in schema", leaving the release in
UpgradeFailed state. Drop the debug block.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit 70a2284551)
Refresh the vendored virt-operator manifest from v1.8.2 to v1.8.4. The
release backports validation of the VMI checksum status fields as a
uint32 range, required on Kubernetes 1.36 where strict CRD numeric
format validation otherwise leaves VMIs stuck in Scheduled. Remaining
changes are bug fixes and CVE remediations; the embedded CRDs and RBAC
are unchanged, so only the operator image tag moves.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 192246e8b6)
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration
concern that rested on an outdated assumption — that an RWO Filesystem
backend-storage PVC pins the VM to its node. KubeVirt has migrated
RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on
migration it creates a fresh target state PVC and copies the small state blob,
so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine
(verified: the VM reports LiveMigratable=True and the copy-on-target PVC is
created on migration). Restore upstream's persistent firmware state for the
windows.* preferences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 59f035ff53)
The vendored seaweedfs chart ships connection pool settings only for the
mysql store, which cozystack does not use. The postgres2 store, which is
enabled, has no pool settings, so the Go sql pool keeps zero idle
connections and every filer metadata lookup opens a fresh PostgreSQL
connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3
operation this adds roughly 2 seconds of latency to every S3 request,
regardless of cluster load.
Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB
object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after
setting these values; pg_stat_activity confirmed connections were
previously recycled every ~200ms and are now reused.
max_open is capped at 40 so that two filer replicas stay below the CNPG
default max_connections of 100.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit 49cbea8c73)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 47638f8569)
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 2b1d3e904d)
## What this PR does
Backport of #3121 to `release-1.5`.
Makes node group `resources` optional, so a node group sized purely by
`instanceType` (with `resources` omitted) passes `values.schema.json`
validation and installs. When a group sets both `instanceType` and
explicit `resources` (both `cpu` and `memory`), the explicit resources
take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt does not
reject a VM that references an instancetype while overriding
`domain.cpu`/`memory`. A partial spec (only one of `cpu`/`memory`)
alongside an `instanceType` fails fast at render time.
Cherry-picked from `a349c753`; conflicts resolved against the
`release-1.5` tree (the management-cluster Talos worker bootstrap stack
present on `main` is not on this branch, so the schema/template/test
layout differ). Generated files (`values.schema.json`, `README.md`,
cozyrds, API types) regenerated with `make generate` to match. `helm
unittest` passes (134 tests), including the four new node-group sizing
cases adapted to this branch's document layout.
### Release note
```release-note
fix(kubernetes): make node group resources optional when instanceType is set
```
PR #2279 made `replication.enabled: true` unconditional so the operator would
always create -primary/-secondary services. But the mariadb-operator validating
webhook (vmariadb-v1alpha1.kb.io) rejects replication when spec.replicas == 1,
so single-replica MariaDB could no longer be created — the HelmRelease never
became Ready.
Re-guard replication on replicas>1 and teach the service-name consumers about
the bare <release> service the operator provisions in both topologies:
- mariadb.yaml: gate `replication` on replicas>1; restore single-replica
external access via `service.type: LoadBalancer` and keep `primaryService`
LoadBalancer for replicas>1 only.
- backup-cronjob.yaml: route single-replica dumps to the bare service
(no -secondary exists without replication).
- dashboard-resourcemap.yaml: always grant RBAC on the bare service, plus
-primary/-secondary when replicas>1, matching the ApplicationDefinition.
- mariadb-rd: list the bare service in services.include so single-replica
installs are visible in the dashboard.
Adds a single-replica e2e test that reproduces the webhook rejection and
guards against regression.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
(cherry picked from commit c7e8f4d975)
The tenant etcd, ingress and monitoring HelmReleases create VM* custom
resources (VMPodScrape directly in etcd/ingress; VMCluster, VMAlert,
VLCluster, VMAlertmanager and VMServiceScrape via the monitoring-system
child of monitoring) guarded by the victoria-metrics-operator validating
webhook with failurePolicy: Fail. On a cold install the operator pod can
be briefly unavailable on :9443, so these releases reconcile before the
webhook backend is serving and are rejected with connection refused. They
carry install.remediation.retries: -1 and would eventually recover, but
the failed-install -> rollback -> retry churn can outlast a cold-install
readiness deadline and surfaces as spurious install failures.
Add a Flux spec.dependsOn from each of these releases to the
victoria-metrics-operator HelmRelease in cozy-victoria-metrics-operator,
so they wait for the operator to be Ready before creating VM* resources.
The system monitoring components gate on the operator the same way, via a
variant-level dependsOn. The cross-shard reference (tenant releases are
sharded, the operator release is not) resolves via helm-controller's
uncached dependency check (APIReader, v1.1.0+).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 6db5ae9c6f)
A node group is sized either by instanceType or by explicit resources, but the
generated values schema required resources on every node group. A node group
with instanceType and no resources — or an empty resources dropped during
typed-CR to HelmRelease translation — failed schema validation with "missing
property 'resources'" even though sizing was fully specified.
Mark resources optional in the node group definition and regenerate the schema,
API types, README, and ApplicationDefinition. The chart template emits
domain.cpu/memory only when resources is present, so a node group sized purely
by instanceType now passes validation and installs.
When a node group sets both instanceType and explicit resources, explicit
resources take precedence: the instancetype is omitted from the rendered VM and
the per-group instancetype lookup is skipped, so KubeVirt accepts the explicit
domain.cpu/memory with no instancetype to conflict with. instanceType stays
required and defaulted (no schema change), so the console's required
instanceType dropdown keeps working — sending the default instanceType plus
explicit resources simply sizes the VM by resources. This replaces an earlier
render-time hard failure on the both-set case, which was unreachable to clear
from the dashboard (the required dropdown cannot be emptied).
Explicit resources are all-or-nothing alongside an instanceType: a partial spec
(only cpu, or only memory) on a group that carries a non-empty instanceType
would emit one domain override next to the instancetype — which KubeVirt rejects
— so it now fails at render time with a clear error. Partial resources on a
group without an instanceType (used to tune kubelet reservations) are unaffected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit a349c753ad)
The seaweedfs COSI driver shipped at v0.1.2 hardcodes read-write S3
actions (Read, Write, List, Tagging) for every BucketAccess and never
reads the accessPolicy parameter. The chart already defines a -readonly
BucketAccessClass carrying parameters.accessPolicy: readonly, but v0.1.2
ignores it, so a credential meant to be read-only can still write
(upload and delete) objects.
Bump the driver to v0.3.1, which honours accessPolicy: readonly by
issuing Read,List-only credentials (read-write and the empty default
stay Read,Write,List,Tagging). The override lives in the package
values.yaml rather than the vendored chart so it survives make update;
the sister extra/seaweedfs chart already runs v0.3.1.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 0c6426feaa)
The cozy-harbor wrapper never passed storageClass to the jobservice
jobLog PVC, so it was created without a storageClassName and the
default-StorageClass admission stamped the cluster default. Because PVC
storageClassName is immutable, this caused rendered-vs-live drift on
upgrade. Preserve an existing PVC's class via lookup and fall back to the
configured storageClass for fresh installs.
The trivy StatefulSet's volumeClaimTemplate had the same latent drift --
.Values.storageClass was forwarded blindly into a VCT that cannot be
patched at all -- so it gets the same lookup-preserve treatment, keyed
off data-<release>-trivy-0.
A live object pinned to an explicit "" (no dynamic provisioning) now
round-trips through upstream's "-" sentinel instead of collapsing to an
omitted storageClassName, and the spec lookup is guarded with
`| default dict` so a PVC carrying no storageClassName key stays omitted.
Adds a helm-unittest regression suite and wires a test target so
make unit-tests stops skipping packages/apps/harbor.
Fixes#2368
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit bdf520bf74)
The s3-service-name.patch renamed the S3 Service to <name>-s3
(seaweedfs-s3), but several consumers still resolved it through
componentName -> <fullname>-s3. name and fullname only coincide when the
release name equals the chart name; the shipped release is
seaweedfs-system, so they pointed at seaweedfs-system-s3 while the real
service stayed seaweedfs-s3:
- the S3 ingress backend and the iceberg ingress backend, which left
ingress-nginx returning 503 on the external S3 endpoint.
The COSI provisioner in-cluster ENDPOINT was wrong on the same axis and
also on the port. Both the s3-gateway branch and the filer-S3 branch
rendered a portless URL that defaults to 443, while the s3 service only
listens on the S3 port (8333); the filer branch additionally targeted
the filer service, which serves no S3. Both in-cluster branches now
resolve seaweedfs-s3 on the configured S3 port.
The default path (s3.ingress.enabled=true) is repaired by the ingress
fix; the COSI in-cluster paths are latent until an operator disables the
ingress. Resolve every consumer through the same name helper the Service
uses, and pin the contract with unittests rendered under the production
release name so the mismatch cannot recur.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 8a14de9abf)
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit ee853b00bd)
entityOperator.topicOperator/userOperator carried no resources, so in tenant
namespaces (the LimitRange defaults containers to 128Mi) the JVM topic-operator
and user-operator are OOMKilled on startup. The entity-operator stays in
CrashLoopBackOff, KafkaTopics/KafkaUsers never reconcile, and on teardown topics
get stuck on the strimzi.io/topic-operator finalizer. Set explicit requests/limits
(256Mi/512Mi) so they no longer inherit the namespace default; add a template test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 1ab5f17cac)
TEMPORARY revert. Restoring persistent EFI/TPM state (4cc60e3) makes
windows.11/2k22/2k25 VMs get a backend-storage PVC that is ReadWriteOnce on the
default StorageClass, which pins the VM to its node and blocks live-migration and
node drains (with evictionStrategy: LiveMigrate this can stall cluster upgrades).
See #3005.
Strip persistent EFI/TPM from the preferences again, but with `yq del` so an
emptied preferredTPM stays `{}` (the v1.8 CRD rejects null) — the bug the original
sed strip caused. Secure Boot and the vTPM stay present, just non-persistent.
Stopgap until vm-state can request an RWX backend (RWX-Block via the upstream
KubeVirt change, or an RWX-Filesystem vmStateStorageClass).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 6ac2601a68)
## What this PR does
Implements the tenant Flux sharding design from #2817: a new
`flux-shard-operator` system package that spreads tenant HelmReleases
across multiple helm-controller shards, so a noisy tenant (e.g. a
HelmRelease stuck in infinite remediation) cannot degrade the others.
- **Placement controller** owns the tenant→shard assignment: the unit of
placement is the tenant (all of its HelmReleases share one shard label),
distribution is greedy least-loaded weighted by HelmRelease count, with
pinned tenants, threshold-gated rebalance, paced moves, and never moving
deleting tenants. Watches are metadata-only, so the controller does not
decode the helm-controller status-patch firehose. Assignments are
recorded on tenant namespaces; HelmRelease labels remain the source of
truth on restarts.
- **Shard runtime** is provisioned by the operator: `shardCount`
helm-controller Deployments cloned from the flux-aio `flux` Deployment
and sanitised, so the image and feature-gates stay version-synced
automatically. flux-aio itself is unchanged and keeps owning everything
without a shard key.
- **CREATE-only mutating webhook** (`failurePolicy: Ignore`) stamps the
tenant's shard onto every HelmRelease at admission, so each one is born
on the correct shard regardless of creation path; any miss degrades
gracefully to the placement controller.
- The hand-rolled `flux-tenants` Deployment is retired: the operator
drains it (backfill `tenants` → `shard<i>`) and deletes it once empty;
migration 44 removes a leftover only under the same guard.
Ships with `shardCount: auto` (the default): the operator sizes the
shard count from the tenant HelmRelease count — `K = clamp(ceil(H/100),
1, min(16, T))` with hysteresis (scale up past 120 HR/shard, down under
60) — so small clusters stay at one shard (≈ today's behaviour) and
large fleets shard out automatically. An integer value pins the count
explicitly.
One deviation from the design doc: the webhook/controller scope is the
`sharding.fluxcd.io/key` label rather than
`internal.cozystack.io/tenantmodule` — every tenant HelmRelease is born
with `key=tenants` (stamped by ApplicationDefinition `release.labels`
and chart templates), so the webhook rewrites it and the placement
controller continuously relabels stragglers; `tenantmodule` covers only
the 6 tenant chart modules.
### Screenshots
Not a UI change.
### Release note
```release-note
feat(fluxcd): tenant HelmReleases are now reconciled by operator-managed helm-controller shards; the flux-shard-operator balances tenants across `shardCount` instances and retires the legacy flux-tenants deployment
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced **flux-shard-operator** to shard helm-controller workloads
with deterministic tenant placement, optional rebalance controls, and
autosized shard count.
* Added a **CREATE-only mutating webhook** to stamp the correct shard
label on tenant HelmReleases.
* **Bug Fixes**
* Application updates now preserve existing **flux-shard** label
assignments to prevent unwanted shard re-stamping.
* **Chores**
* Added platform packaging for the new component, including an image
build step.
* Removed the legacy tenants deployment and added a migration to retire
it safely.
* **Tests**
* Expanded unit and chart tests for placement, provisioning, webhook
behavior, and configuration parsing.
* **Documentation**
* Added operator documentation describing configuration and metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
During node reprovisioning in tenant Kubernetes clusters, the VMI (and
its hot-plug pod) can outlive the parent VirtualMachine. The upstream
kubevirt-csi-driver treats a missing VM as "detach succeeded", so the
hot-plug pod stays alive and keeps the infra storage device exclusively
attached to the source host — every subsequent attach of that volume
fails, on DRBD with `failed to set source device readwrite`. This was
hit repeatedly in production tenant clusters (see
kubevirt/csi-driver#182).
This PR:
- updates the pinned `kubevirt.io/csi-driver` module to the latest
upstream commit;
- backports kubevirt/csi-driver#184 into the controller wrapper: after
the upstream unpublish returns success, the wrapper re-checks
`VMI.status.volumeStatus` and, if the hot-plug is still reported there
(e.g. because the parent VM is already gone), detaches it from the VMI
directly via the `removevolume` subresource and waits until it
disappears.
The backport is self-contained in the wrapper and can be dropped once
kubevirt/csi-driver#184 merges upstream and the module pin is bumped
past it.
Credit to @mattia-eleuteri for the root-cause analysis and the upstream
fix.
### Screenshots
Not a UI change.
### Release note
```release-note
fix(csi): tenant CSI driver no longer leaves orphan hot-plug volumes exclusively attached to the source host when the VMI outlives its deleted VirtualMachine during node reprovisioning
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Enhanced hotplug volume unpublishing with automatic verification to
ensure volumes are fully detached before completion, including retry
logic with up to 2-minute timeout.
* **Dependencies**
* Updated KubeVirt API and CSI driver dependencies to latest versions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The shardCount description and the shard.resources value overflowed their columns, pushing the internal separators out of line. Reflow every column to its widest cell so the raw table renders cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
Consolidated landing branch for a set of interdependent platform/CI
stabilization fixes. It began as the source PRs in the table below (in
dependency order) and has since grown with review-driven fixes and a few
production-behavior changes — the full current inventory is under
**Scope** below. Landing them as one unit lets CI run against the real
combined state instead of a fragile bottom-up merge train.
## Source PRs (dependency order)
| # | Commit | Source PR | Addresses |
|---|--------|-----------|-----------|
| 1 | fix(capi): startupProbe on capi-controller-manager | #2946 | capi
crashloop during cert provisioning |
| 2 | test(e2e): bucket.bats port-forward + S3 client reliability |
#2944 | flaky bucket test |
| 3 | fix(seaweedfs): restore -lock BucketClass, s3 svc name, drop
volumeSizeLimitMB | #2943 | missing bucket/access classes |
| 4 | test(e2e): in-cluster Cilium endpoint-leak healer (install + apps)
| #2874 | cilium "IP already in use" leak |
| 5 | refactor(build): mode=max registry cache + main-only warmer |
#2938 | build cache |
| 6 | ci(build): isolate each PR build on its own ephemeral runner VM |
#2939 | shared-buildkit contention |
| 7 | fix(e2e): LINSTOR post-install waits on a single 15m deadline |
#2928 | LINSTOR provisioning timeouts |
| 8 | test(metallb): assert digest-pinned image form, not version
literal | #2873 | brittle metallb assertion |
## Why consolidated
These form a dependency DAG (verified from CI logs): e.g. 2943 needs
2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested
bottom-up, each lower PR runs with none of the fixes above it and can't
go green alone. This branch carries all of them, so CI runs against the
real combined state.
## Verification status
CI is green on the latest head (`5653c30`): full-suite E2E passes
end-to-end and is reproducible (2 of 3 runs on this SHA green). The one
red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the
sandbox, unrelated to the diff — a different environmental subsystem
failed each run (details in the comments below). The earlier 3-hour
crust-gather snapshot hang is fixed in `5653c30`.
## Scope beyond the original 8
The branch has grown past the 8 source PRs above with review-driven
fixes and a few production-behavior changes. For reviewer transparency,
the full set:
**Review fixes (@lexfrei review):**
- **B1 (blocker)** — `8f41910` converts the SeaweedFS
`-lock`/`-readonly` BucketClasses and the s3 service-name override into
`patches/`, wired into `make update`, so a re-vendor no longer drops
them.
- **FU1** — `cc99230` grants the cilium leak-healer `delete` on pods
(both delete remedies were RBAC-forbidden).
- **FU3** — `014de3f` adds a unit test for the HelmRelease update
conflict-retry path.
- **FU4** — `6deddc6` makes the leak-healer refuse a disconnect for any
non-terminal owner phase, not just `Running`.
- **FU7** — `f3914d2` sets `persist-credentials: false` on the
build-cache checkout (SHA-pinning is handled repo-wide in #2849).
**Production-behavior changes (not in the table above):**
- VPA `updateMode: Auto → Initial` for etcd (`6e9ff90`) and monitoring
(`0997105`), plus `vmselect`/`vmstorage` `minAllowed` floors — stops
install-time eviction churn.
- cozystack-api: `retry.RetryOnConflict` on the Application→HelmRelease
update path (`728f676`).
**E2E robustness follow-ups:** per-test crust-gather snapshots
(`50e5f94`), EtcdBackupSchedule wait (`763b85e`), tenant-node
single-deadline wait (`9e43d45`), harbor BucketClaim 10m budget
(`d7f244f`), tenant API via LoadBalancer (`9beda4e`), crust-gather pin
(`95f42aa`).
**Not addressed (by decision):** FU2 — the s3 Service keeps
`seaweedfs.name`-based naming via patch (rendered output unchanged;
cozystack renders `fullname == seaweedfs`, so it matches the
`componentName`/`fullname` siblings in practice). FU6 — fork PR build
push is handled by the existing mirror-to-same-repo-branch workflow.
## Relates to
#2946#2944#2943#2874#2938#2939#2928#2873 — if this lands, those
can be closed; otherwise they remain the granular per-PR review path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added CI build cache warming for main branch builds.
* Added a best-effort in-cluster Cilium endpoint self-heal watchdog for
e2e installs.
* Extended SeaweedFS COSI with object-lock support plus separate
readonly access.
* **Improvements**
* Reduced e2e install flakiness with a shared readiness deadline, safer
waits, and improved cleanup/diagnostics capture.
* Prevented install-time churn by switching VPA update mode to
**Initial**.
* Added container startup probes and updated SeaweedFS volume sizing.
* **Tests**
* Improved e2e robustness (etcd backup schedule waiting, digest-pinned
image checks, and corrected S3 port-forwarding).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds an operator-facing way to serve platform services and the root
tenant's ingress under a pre-existing wildcard TLS certificate instead
of minting per-host ACME certificates. Implements the "native
references" design decided on #2812.
A single value, `publishing.certificates.wildcardSecretName`, names a
TLS Secret the operator has already created in the publishing namespace
(`tenant-root` by default). Only the name travels through the
`cozystack-values` channel — the certificate and private key are never
broadcast, unlike the public `kube-root-ca` that already rides it.
When set:
- **ingress-nginx path** (`gateway.enabled=false`): the publishing
tenant's ingress controller serves the Secret as its
`--default-ssl-certificate`, and the platform service Ingresses drop
their cert-manager cluster-issuer annotation and `tls` block. No
per-host ACME.
- **Gateway API path** (`gateway.enabled=true`): a new `TenantGateway`
cert mode, `existingSecret`, references the Secret in the wildcard/apex
listener `certificateRefs` and mints no Issuer or Certificate. Switching
into this mode garbage-collects any ACME machinery left from
`http01`/`dns01`.
Scope is the root tenant — everything is same-namespace, so no RBAC or
NetworkPolicy changes. Extending wildcard mode to child tenants needs
the Secret replicated into each tenant namespace and is left as a
follow-up.
Closes#2812. Part of #2811.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
feat(platform): operators can serve platform services and the root tenant's ingress under a pre-existing wildcard TLS certificate via `publishing.certificates.wildcardSecretName`, instead of minting per-host ACME certificates
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added an "existingSecret" TLS certificate mode and a wildcardSecretRef
option for TenantGateway and platform values to use a pre-existing
wildcard TLS Secret.
* **Platform / Templates**
* Introduced operator "wildcard secret" mode across Gateway and many
Helm charts (Ingress, apps) to skip per-host ACME provisioning and
optionally wire a default SSL cert.
* **Tests**
* Added extensive chart and controller tests covering existingSecret
behavior and cert-mode transitions.
* **Documentation**
* Updated gateway docs describing the new wildcard secret mode and
requirements.
* **Chores**
* Added Helm unit test makefile targets to several packages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The restored COSI -lock/-readonly BucketClass/BucketAccessClass and the s3
Service name override lived only as raw edits in the vendored chart tree, so
the next `make update` (rm -rf charts && curl | tar && patch) would silently
drop them again — the same way the 4.31 bump did. Capture both as patch files
under patches/ and wire them into the Makefile update target, matching the
existing s3-tls-main-port / cosi-provisioner-sa-name customizations. No
rendered output changes; this only makes the customizations survive re-vendor.
Addresses the B1 review blocker on #2948.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
The crust-gather install.sh substitutes the version verbatim into both
the release-tag path and the asset filename:
releases/download/<ver>/kubectl-crust-gather_<ver>_<target>.tar.gz
The release tag is 'v0.15.2', so passing '0.15.2' 404s on the tag path
(curl exit 22) and fails the image build. The 'v0.15.2' asset variant
exists too, so the leading v satisfies both. Verified HTTP 200.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.
run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.
Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Auto mode evicts running etcd members to apply VPA recommendations, churning the StatefulSet (disruptive to quorum and a Cilium IP-reuse-leak trigger under load). Initial sizes at pod creation without evicting; the existing 250m/256Mi minAllowed floor covers the no-runtime-resize tradeoff. Same eviction-churn class as the monitoring VPA fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths).
Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds `spec.timeout: 10m` to the `kubevirt-csi-node` HelmRelease template
in `packages/apps/kubernetes/templates/helmreleases/csi.yaml`, aligning
it with every other tenant addon HR in the same directory (cilium,
coredns, ingress-nginx, etc. — all set `timeout: 10m`). `csi.yaml` was
the only one missing it; commit 9f9d8f85 ("Align timeouts for
HelmReleases") missed this template.
## Symptom
E2E NFS test times out waiting for `nfs-test-pod` to reach `Succeeded`.
Cozyreport events show the tenant `csi` HR running an Install →
Uninstall → Install remediation cycle. With
`install.remediation.retries: -1` and the default 5m Helm timeout,
kstatus marks the `kubevirt-csi-node` DaemonSet `InProgress` past 5m on
a slow tenant control-plane bringup (Kamaji APIs flapping, VM image
pulls), helm-controller times out, the remediation uninstalls the
DaemonSet, then reinstalls. Any volume mount in flight at that moment is
wedged.
## Origin
Single commit lifted unchanged from #2619.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the Container Storage Interface (CSI) Helm release settings to
include a `spec.timeout` of **10 minutes** when the etcd namespace value
is set (keeping the existing `spec.interval` at **5 minutes**).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add `rm -rf charts/prometheus-operator-crds` before `helm pull` in the
update target so stale files from previous chart versions don't linger
across updates. This matches the convention used by cilium,
external-secrets-operator, external-dns, and cert-manager Makefiles.
Address review feedback from gemini-code-assist on
packages/system/prometheus-operator-crds/Makefile:9.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart previously stripped the bundle to 4 CRDs (ServiceMonitor,
PodMonitor, PrometheusRule, Probe) — the set VictoriaMetrics operator
needs for service discovery. This was correct when the chart was
vendored inside system/victoria-metrics-operator (commit ba5c785cc,
2023), but the prune was carried over when the chart was extracted
to its own package, and again when the CRDs were propagated into
tenant Kubernetes clusters (commit 1e2b66131, 2025).
In tenant clusters the partial set blocks third-party apps that ship
their own prometheus-operator (e.g. kube-prometheus-stack) — the
missing CRDs (Prometheus, Alertmanager, AlertmanagerConfig,
PrometheusAgent, ScrapeConfig, ThanosRuler) prevent the operator from
deploying its instances against the platform-managed CRD set.
Drop the rm -f line in the Makefile so the full upstream bundle ships,
and pin --version 25.0.0 to keep the operator CRD version at v0.87.0
(operator version bump is a separate concern, out of scope here).
Tenant apps can now set crds.enabled: false on their kube-prometheus-
stack release and consume the platform CRDs directly.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Aligns kubevirt-csi-node HR with every other tenant addon HR in
packages/apps/kubernetes/templates/helmreleases/ (cilium, coredns,
ingress-nginx, etc.) — all set timeout: 10m, csi.yaml was the only
exception. 9f9d8f85 ("Allign timeouts for HelmReleases") missed it.
Symptom: e2e NFS test (hack/e2e-apps/run-kubernetes.sh) times out
waiting for nfs-test-pod to reach Succeeded. Cozyreport events show
the tenant csi HR running Install -> Uninstall -> Install remediation
cycles. With install.remediation.retries: -1 and the default 5m Helm
timeout, kstatus marks the kubevirt-csi-node DaemonSet 'InProgress'
past 5m on a slow tenant control-plane bringup (Kamaji APIs flapping,
VM image pulls), helm-controller times out, the remediation uninstalls
the DaemonSet, then reinstalls. Any volume mount in flight at that
moment is wedged.
Observed on PR #2619 CI run 26142986969: csi HR remediation at
~07:07 happened ~10 min into the nfs-test-pod wait, killing the
mount and timing out the pod-Succeeded condition.
10m matches the surrounding HRs and gives the DaemonSet headroom
without changing remediation semantics on truly-broken installs.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 866588ff61)
Myasnikov Daniil