## What this PR does
Aligns the vendored CloudNativePG chart metadata and CRDs with the
already-pinned 1.27.3 operator image. CNPG 1.27.3 stores the
instance-manager session identifier in
`Backup.status.instanceID.sessionID`; the 1.27.1 CRD prunes that field,
causing false instance-manager restart errors and failed backups.
Fixes#3479.
### Screenshots
Not applicable.
### Release note
```release-note
fix(postgres-operator): align CloudNativePG CRDs with the pinned 1.27.3 operator image so PostgreSQL backups no longer fail with false instance-manager restart errors
```
## What this PR does
Hand-backport of #3138 to `release-1.5`. The `backport` label was
applied at merge time but no backport PR was ever opened, so the change
never reached this branch — found by auditing the branch with
`cmd/backport-audit`, which reported it as `MISSING`. Like the other
items in that set, it merged before #3155 fixed the bot's
`conflict_resolution` input, so the conflicting cherry-pick was dropped
with no draft and no failing check.
Velero binds its metrics/health endpoint (`:8085`) only after loading
plugins and connecting to the API server. Under a heavy parallel
platform install the upstream liveness probe starts checking at
`initialDelaySeconds=10` and kills the container after
`failureThreshold=5`, so velero BackOff-restarts and never escapes the
loop inside the install-gate window. Because `backupstrategy-controller`
hard-depends on velero, this blocks the gate. The fix adds a
`startupProbe`, which holds liveness and readiness off entirely while it
is failing, granting `30 * 10s = 300s` of startup grace and then handing
over to the unchanged upstream liveness for steady-state crash
detection.
### Backport adaptations
This branch vendors velero chart **11.0.0** (appVersion 1.17.0) rather
than `main`'s 12.0.3 (1.18.1), so three things differ from the original.
All are recorded in the commit message as well.
- **The chart version is not bumped.** Only the patch re-application is
added to `update`. The `helm pull` here was previously unpinned, which
would have let `update` float to a chart the patch does not apply to, so
it is pinned to `11.0.0` — the version already vendored. That makes the
new patch step deterministic and leaves the vendored tree
byte-identical.
- **The `test:` target is added.** It exists on `main` but not here, so
without it the shipped regression guard would never run.
- **The image assertions pin this branch's own versions** (velero
`v1.17.0`, `plugin-for-aws` `v1.12.1`, kubevirt plugin `v0.8.0`), and
the `upgrade-crds` rationale is restated for chart 11.0.0, where that
Job still runs a kubectl image — `main`'s comment about the Job running
the velero image natively is true for 12.x only.
The `startupProbe` hunk itself applies to chart 11.0.0's deployment
template unchanged, in the same position inside the `metrics.enabled`
gate.
### Verification
`helm unittest .` passes 5/5 in `packages/system/velero`. The assertions
are mutually corroborating rather than vacuous: the `equal` assertions
prove the `startupProbe` path actually renders, and the same paths are
asserted absent under `velero.metrics.enabled: false`, proving it
follows the same gate as liveness and readiness so disabling metrics can
never leave a `startupProbe` pointed at an unbound port.
### Screenshots
Not applicable — no UI change.
### Release note
```release-note
fix(velero): add a startupProbe so a slow velero server start under heavy parallel install no longer crashloops the container and blocks the install gate
```
## What this PR does
Hand-backport of #3034 to `release-1.5`. The `backport` label was
applied at merge time but no backport PR was ever opened, so the change
never reached this branch — found by auditing the branch with
`cmd/backport-audit`, which reported it as `MISSING`.
Every candidate that went missing this way merged before #3155 landed,
when `conflict_resolution` was passed as a top-level input instead of
nested under `experimental`. The action ignored it and fell back to
`fail`, which on a conflicting cherry-pick opens no PR and reports no
failure, so these were dropped with no draft to find and no red check to
notice.
The six original commits are cherry-picked with `-x`, unchanged and in
order. No adaptation was needed: the cherry-pick was clean, and the net
diff is identical to the change on `main`.
### Verification
The two `.diff` files are image patches applied at build time against
upstream `container-object-storage-interface`, so what matters is that
this branch pins the same upstream source. It does — both `release-1.5`
and `main` set `ARG COMMIT_REF=v0.2.2` in
`images/objectstorage/Dockerfile`, and the resulting patch set on this
branch (`91`, `92`, `93`) is byte-identical to `main`'s. Patch `91` was
already present here unchanged, so `92` applies over the same base it
does on `main`.
The branch differs from `main` only in its builder pins (`golang:1.24`
versus `1.26`, unpinned base digests), none of which affect patch
application.
### Screenshots
Not applicable — no UI change.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
The tenant.cozystack.io/<ancestor> namespace labels drive the
<ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor
tenant reach a descendant namespace only when the descendant carries
tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by
splitting .Release.Namespace, which only reached one level up: a direct
child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev
under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely.
The root ingress controller in tenant-root then could not reach nested
tenants, surfacing as upstream timeouts.
Derive the chain deterministically from the namespace name instead. Every
tenant descends from tenant-root, and tenant.name builds each child as
<parent-namespace>-<word>, so every progressive dash-prefix of the name is
a real ancestor namespace. This needs no cluster lookup, renders offline
(so it is unit-testable), and converges on every reconcile regardless of
parent/child HelmRelease ordering. The ownerReference lookup is made
non-fatal so the template still renders when the parent namespace lookup
is empty; in a real reconcile the parent always exists and the
ownerReference is always emitted.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 1ac9398a4b)
(cherry picked from commit 6ce09079b7b84ed124268addfd21881d64976541)
(cherry picked from commit ac7176257b517cbb09fc1792f90ce1f2a18d89be)
Velero binds its metrics/health endpoint (:8085, http-monitoring) only
after the server loads its plugins and connects to the API server. Under
a heavy parallel platform install that can take minutes, but the upstream
liveness probe begins checking /metrics at initialDelaySeconds=10 and
kills the container after failureThreshold=5 (~130s total) before :8085
is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it
BackOff-restarts, and never escapes the loop inside the install-gate
window. Because backupstrategy-controller hard-depends on velero, this
blocks the gate.
Add a startupProbe to the velero server container. While it is failing
the kubelet holds off liveness and readiness entirely, granting
failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then
hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for
steady-state crash detection. This is strictly better than loosening
liveness, which would also blunt steady-state detection.
The upstream velero chart (12.0.3) does not render a startupProbe, so the
package patches the deployment template via patches/add-startup-probe.patch
(re-applied after re-vendor by the Makefile update target, matching the
seaweedfs/ingress-nginx convention) and supplies the startupProbe through
the package values override. Pin the startupProbe budget and the unchanged
liveness with a helm-unittest regression guard.
Backport adaptations for release-1.5, which vendors velero chart 11.0.0
(appVersion 1.17.0) rather than main's 12.0.3 (1.18.1):
- The chart version pin stays at the version this branch already
vendors; only the patch re-application is added to `update`. The
`helm pull` was previously unpinned, which would have let `update`
float to a chart the patch does not apply to, so it is pinned to
11.0.0 to keep the new patch step deterministic.
- Added the `test:` target, which exists on main but not here, so the
shipped regression guard actually runs.
- The image assertions pin this branch's own vendored versions
(velero v1.17.0, plugin-for-aws v1.12.1, kubevirt plugin v0.8.0),
and the upgrade-crds rationale is restated for chart 11.0.0, where
that Job still runs a kubectl image.
The startupProbe hunk applies to chart 11.0.0's deployment template
unchanged, in the same position inside the metrics.enabled gate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit f7271b8011)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
While the backend Bucket is not ready the controller returns an error to
force a requeue, but that error is only logged at klog.V(3) and is
suppressed at default verbosity, so an operator cannot see why the claim
is stuck. Record a WaitingForBucket Warning event on the BucketClaim on
each requeue (event aggregation folds the repetition) and still return
the error so the requeue happens as before. Pin the requeue contract in
the test by asserting the error carries the stable "not ready yet"
substring, not merely that it is non-nil.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 5991808d28)
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 0855ff2d1b)
The objectstorage-controller package ships the COSI control plane; its commits
use the objectstorage-controller scope, which had no area mapping and fell back
to area/uncategorized. Map it to area/storage alongside the other COSI/storage
scopes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 4f5dcea432)
The bucket suite waited on '{.status.bucketReady}' without an expected value;
a bare jsonpath match is satisfied by the literal "false", so the wait passed
against an unready claim. Both suites also allowed 300s/600s, long enough to
tolerate the propagation race rather than catch it, so they stayed green
whether or not the BucketClaim ever converged.
Assert bucketReady=true on a 120s bound. Convergence is now controller-driven
(the controller requeues the claim until the backend Bucket's readiness
propagates) and completes within tens of seconds; a claim frozen by the
propagation race never converges and now fails the wait fast instead of being
masked.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 63d14e7487)
The controller and the per-driver sidecar both run single-replica, so leader
election provides no high availability -- it only governs how quickly a freshly
recreated pod takes over. With the upstream 150s lease duration a recreated
provisioner pod stalls reconciliation for up to 150s while it waits out the
stale lease still held by the previous, now-deleted pod. Lower the lease timings
to the standard Kubernetes values (15s/10s/2s) so a recreated pod resumes
reconciliation in roughly 15s.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 250d92323d)
The dynamic-provisioning path re-reads the live Bucket after create and copies
its readiness onto the BucketClaim, but only during a reconcile. Nothing
re-drives that reconcile afterwards: the controller does not watch Buckets, and
the generic controller drops no-op resync deltas (reflect.DeepEqual), so once
the BucketClaim stops changing the listener is never re-invoked. The Bucket's
false->true transition, performed asynchronously by the sidecar on a different
object, is therefore reflected onto the claim only if the Bucket happened to be
ready during that single reconcile -- a race the sidecar usually loses, leaving
bucketReady frozen at false and BucketAccess never granted.
Return an error while the backend Bucket is not ready so the work queue
requeues the claim with exponential backoff; each retry re-reads the live
Bucket until its readiness has been propagated, converging deterministically
instead of depending on a resync that never fires.
Add a unit test pinning the contract (requeue while not ready, propagate true
once ready) and adjust the existing status-update conflict-retry test, which
pinned the old never-requeue behavior.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 4437cccaf0)
The operator image is pinned to 1.27.3 while the vendored chart and
CRDs still advertise 1.27.1. Kubernetes prunes Backup
status.instanceID.sessionID, causing false instance-manager restart
failures.
Import the official 1.27.3 CRDs, align the chart appVersion, and preserve
the overlay across make update. Add render and API round-trip regression
coverage.
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Backport of #3359 onto release-1.5.
The cert-manager webhook listened on 10250, the port the kubelet also
serves. When a connection to the webhook Service resolved to a node IP
rather than a Pod IP it reached the kubelet, which completed the TLS
handshake with its own node serving certificate, and the API server then
rejected every cert-manager admission call cluster-wide:
failed calling webhook "webhook.cert-manager.io": ... x509: certificate
is valid for srv3, not cert-manager-webhook.cozy-cert-manager.svc
The certificate is valid — it is simply the wrong server's — so the error
reads as a cert-manager fault and hides the misroute that caused it.
Moving the webhook to 10260 is upstream's own remedy for this signature.
It does not fix the misroute; it removes the collision that turns a rare
transient one into a cluster-wide outage attributed to the wrong
component.
The override lives in the package-root values file, which `make update`
leaves alone, so it survives re-vendoring.
Stacked on the #3199 backport (#3202): release-1.5 ships an EMPTY
packages/system/cert-manager/values.yaml, so every resource override in
that file arrived after v1.5.2. The bot's cherry-pick of this fix
conflicted for that reason alone — nothing to merge into — and applies
cleanly once the cainjector backport is in place.
Upgrade safety is unchanged from the original: one replica, no explicit
strategy, so maxUnavailable 0 / maxSurge 1 apply and the replacement Pod
must be Ready before the old one goes. The Service targets a NAMED port,
so during the overlap each Pod is reached on the port it actually serves
and there is no window with zero ready endpoints.
Closes#3355
(cherry picked from commit 3affec9920)
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Backport of #3344 onto release-1.5.
The bot's cherry-pick imported hack/e2e-chainsaw/mariadb/chainsaw-test.yaml
whole, because release-1.5 has no such file: the app suite here is still
BATS, and the Chainsaw port landed with #2826 on main only. That file is
dropped and its widening applied to the gate that actually runs on this
branch instead.
That gate needed the change more than the chainsaw one did. In
hack/e2e-apps/mariadb.bats both tests wait 80s for an endpoint address to
appear, and an address appears only once a replica has cleared its startup
then readiness probe. This commit raises the startup budget to 310s, so an
80s ceiling would fail runs the probe was still willing to wait for --
importing the chainsaw file and leaving the BATS suite alone would have
turned the fix into an e2e failure. Both waits go to 600s, matching the
10m the upstream chainsaw assert uses and for the same reason.
packages/apps/mariadb/Makefile gains the `test:` target. It is absent on
release-1.5 -- it arrived with #3071, which was not backported -- and
hack/helm-unit-tests.sh discovers suites by probing for that target, so
without it the backported tests/startup_probe_test.yaml would never be
executed by `make unit-tests`.
Verified locally: 4 assertions in tests/startup_probe_test.yaml pass.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit a37715062e5da24e54956f075dd2d6417c4dff2e)
Address review feedback on the repeat in-place restore fix:
- Make the freshness discriminator strict (creationTimestamp > StartedAt) so
an exact tie resolves to "not fresh" and the caller purges, matching the
function's stated conservative default; a fresh Cluster is always created
well after StartedAt so this never misclassifies the status-write-race path.
- Document why a wall-clock timestamp comparison was chosen over recording the
purged Cluster's UID on the TargetPurged condition (the UID approach leans on
the same status write the race path assumes can fail).
- Add a reconcile-level regression test that drives reconcileCNPGRestore end to
end with a fake client: a stale leftover recovery Cluster from a prior restore
is purged, and a Cluster this restore just re-created is not re-purged. This
covers the actual call site where #3311 lived, not just the pure helpers.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit a8494d874f)
(cherry picked from commit 0fdf3b5a6152db8912a094493f2ff5ea122ac7bb)
(cherry picked from commit 2d6083ccb6f418a35ba1ae058b3a91f7edcc89fd)
cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs
into its informer caches at startup. On a full cozystack install that working
set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is
OOMKilled during cache population and CrashLoops before it injects any
caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle
and every admission call fails with "x509: certificate signed by unknown
authority" (failurePolicy: Fail), intermittently blocking Ingress creation
and stalling tenant installs.
Raise the cainjector memory limit to 512Mi (matching the controller) and the
request to 128Mi, and add a helm-unittest that pins the sizing so a blanket
resource change or a `make update` cannot silently lower it again.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit c3afc63cb3a84c8085e991223ebd3f18a70d5183)
(cherry picked from commit 9f312d7a216c34724848142415d2de7c086a711e)
A repeat in-place PostgreSQL RestoreJob into an already-restored target
silently no-opped: the destructive purge was skipped because the live
Cluster still carried spec.bootstrap.recovery left over from the previous
restore, and the re-purge guard mistook that for a freshly-recovered
Cluster produced by this restore's own purge. The job then reported
Succeeded while the PVC, disk, and data were never touched.
Tell the two apart by the Cluster's creationTimestamp relative to the
RestoreJob's StartedAt: only a recovery Cluster created at/after StartedAt
is treated as this restore's own and skipped; one that predates StartedAt
is leftover from an earlier completed restore and is purged so the restore
re-bootstraps from the backup.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit cf1b6ba28e)
(cherry picked from commit c85c9f38a42765d7da052e8c4b5ed7ab2aef5c7b)
(cherry picked from commit 507a48762292665343195da99913d64d4b256906)
Backport of #3339 onto release-1.5, on top of the #3282 adopt-in-place
fix (#3326).
Three differences from main, all forced by what release-1.5 does not
carry:
* The repair migration is renumbered 53 -> 45 and stamps 46, since
release-1.5 tops out at migration 44 with targetVersion 45.
platform values.yaml goes to targetVersion 46.
* Migrations 43 and 45 keep release-1.5's inline labelled stamp
instead of lib/cozystack-version.sh. That helper is the shared
stamp refactor (#2980), whose backport (#3065) was closed, so it
does not exist on this branch. Only lib/seaweedfs-db-adopt.sh --
the part that actually carries the fix -- is sourced.
* The extra/seaweedfs post-delete cleanup hook and its three tests
are dropped. The hook arrived with #3092, a breaking change that
was never backported, so there is nothing here for #3339's changes
to it to apply to. Comments in lib/seaweedfs-db-adopt.sh and the
recovery runbook that promised the hook would reclaim an orphaned
volume are corrected to say it does not on 1.5.x.
The migration renumbering is carried through the shared helper, the
bats suite (STAMP 54 -> 46) and the runbook prose.
Verified locally: helm unittest green for both seaweedfs charts (51 + 16
assertions) including both blind-upgrade guard canaries, and 34 bats
assertions across migration-seaweedfs-db-adopt, seaweedfs-naming-audit
and seaweedfs-guard-parity.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 88e9fdcbbbff8aeab317b08557ca6bd6c8b5706d)
Classify each tenant from its volume PVCs, re-bind the volumes of a tenant
installed fresh on 1.5.x so the adopted names pick them up, stop a split cluster
before upgrading it, and clear the leftovers the upgrade cannot remove itself.
The recovery steps select the renamed set by a precise filter and derive secret
names from the fullname helper, so they work for an instance running under a
non-default name and for pooled or zoned volumes.
Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 088669fb5ae5106aa1d4d22e3226488f557e4b02)
With the chart-based names pinned back, an upgrade adopts the running workloads
and their volumes in place. Before 4.31 the chart ignored the release name, so
this holds for an instance running under any name: its data sits on the
chart-named data1-seaweedfs-volume-* PVCs either way.
Two states cannot be adopted and stop the render instead, each pointing the
operator at the matching recovery step:
- Fresh on 1.5.x: only the renamed volumes exist, so adopting the chart-based
name would rename the workloads away from that data and bring up an empty
cluster. Helm cannot move data between PVCs, so the operator is sent to the PV
re-bind procedure.
- D-split: both sets exist AND the renamed volume servers are live, so they may
hold objects written through the split endpoint. Adopting would strand them, so
the operator is sent to reconcile the split first. A duplicate that never served
(zero ready replicas) is safe to adopt and renders through.
The guard mirrors the chart's fullname helper to recognise what 4.31 named this
instance (the helper appends the chart name when the release name does not
contain it, then truncates), and matches the renamed volumes by shape plus the
chart labels on the StatefulSet rather than reconstructing the exact name.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 052ce2e4a3)
(cherry picked from commit 5a724e0bc0d43c8afb621a6da4aab5b13a86e774)
Upstream chart 4.31 switched resource names from the chart name to the release
name. The data-plane release is <name>-system, so every StatefulSet was renamed
to <name>-system-*. StatefulSet names are immutable, so the upgrade could not
rename in place: Helm stood up a second, empty set beside the running one while
the data stayed on the original data1-seaweedfs-volume-* PVCs. Where the node
count let both sets run, they shared one S3 Service and one filer metadata
database, so the empty set answered reads and mixed volume IDs from an
independent sequence into shared metadata.
Pin fullnameOverride so the rendered names no longer follow the release name,
matching the convention already used by other system packages (cozy-proxy,
flux-operator, linstor-scheduler, victoria-metrics-operator).
The s3-consumer suite pinned the name/fullname divergence this override removes
by default, so it now re-creates that divergence explicitly and keeps guarding it
regardless of the shipped default.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 7bcd8e07bb)
(cherry picked from commit 1da45ead8262334636919c097019afdf71da412e)
## What this PR does
Adds the flux CLI to the release-1.5 cache warmer, which #3469 moved
onto the ephemeral runner shape without carrying over the toolchain step
that main's warmer and this branch's `pull-requests.yaml` both have.
`make build` finishes at `packages/core/installer`, whose
`image-packages` target shells out to `flux push artifact`. The
self-hosted runner the warmer used to target had flux baked in; the
ephemeral shape does not.
Confirmed on the first warmer run after #3469 merged ([run
30430753152](https://github.com/cozystack/cozystack/actions/runs/30430753152)):
it failed at 49 minutes with `/bin/sh: 1: flux: not found` on
`Makefile:35`, the last of 28 packages.
The cache itself came out warm, because every package exports its cache
as it builds and installer is last:
| | |
| --- | --- |
| cache export operations | 87 |
| cache manifests written | 70 |
| repos created under `CACHE_REGISTRY` | 34 |
| `ubuntu-container-disk` refs (`v1.30`–`v1.35`, `mode=max`) | all 6 |
| real auth or export failures | 0 |
So the only loss was the `cozystack-packages:release-1.5` artifact push,
and nothing downstream depends on it — PR builds build the installer
themselves with flux present. This also settles the open question from
#3469: the registry credentials can create repositories under the new
`cozystack-cache-release-1.5/*` namespace, so no fallback to a cache-tag
suffix is needed.
Fixing it still matters. A permanently red warmer means a genuine
breakage is indistinguishable from this known-broken tail, and nothing
else reports on that workflow.
The alternative would be teaching the warmer to run the 28 image builds
without the artifact push, but no such target exists and it would
diverge from main's warmer for no gain. Installing the CLI keeps the two
warmers identical in shape, and matches main publishing
`cozystack-packages:main` from its own warmer.
### Release note
```release-note
NONE
```
`make build` finishes at packages/core/installer, whose image-packages
target shells out to `flux push artifact`. The warmer used to target the
self-hosted runner, which had flux baked in; #3469 moved it to the
ephemeral shape without carrying over the toolchain step that main's
warmer and pull-requests.yaml both have for exactly this reason.
Confirmed on the first run after #3469 merged (run 30430753152): it failed
at 49 minutes with `/bin/sh: 1: flux: not found` on Makefile:35, the last
of 28 packages. Everything before it had already exported its cache -- 87
export operations, 70 cache manifests, 34 repos created under
CACHE_REGISTRY, including all six ubuntu-container-disk refs -- so the
cache came out warm and only the cozystack-packages artifact push was
lost. A red warmer is still a broken signal, and nothing else would have
reported it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Nothing has ever warmed `release-1.5`'s build cache, so every PR build
on this branch runs cold end to end and exceeds its 30-minute timeout.
This branch's copy of `build-main.yaml` is dead code. It triggers on
`push: branches: [main]`, which never happens here, and it targets
`[self-hosted]` — the decommissioned cozy-runner-1 (#2937) — so it could
not have run even if it fired. This PR points it at this branch instead.
Both halves of the cold build trace back to 39cdc8866
(`feat(kubernetes): bootstrap tenant workers with Talos via CABPT`)
landing on `main` on 2026-06-16, six days before `release-1.5` was cut
on 2026-06-22:
- That commit dropped `ubuntu-container-disk` from `make image`, so
`main` stopped writing its six `:<minor>-buildcache` refs. `release-1.5`
is now the only branch that still builds the image, and the CI log shows
all six importers failing with `not found`. Serially that was ~28min of
the 30min budget; #3457 parallelises the target down to ~10min.
- Every other cache ref carries `main`'s sources, which have diverged
from this branch, so most of the remaining 28 packages miss too.
Measured on the #3457 run: ~38min across the full build, and no single
hot spot left to attack — `apps/kubernetes` at 15m07s and
`system/linstor` at 5m39s, everything else 20-60s each.
### Changes
| | before | after |
| --- | --- | --- |
| trigger | `push: [main]` | `push: [release-1.5]` |
| runner | `[self-hosted]` (decommissioned) |
`oracle-vm-24cpu-96gb-x86-64` |
| cache namespace | shared `cozystack/` | `cozystack-cache-release-1.5/`
|
| `IMAGE_TAG` | `main` | `release-1.5` |
| warmer timeout | 90 | 120 |
`pull-requests.yaml` gets the same `CACHE_REGISTRY` so PR builds read
what the warmer writes, and its Build timeout goes to 75 minutes.
The cache namespace is branch-scoped on purpose. Writing the shared refs
from here would replace `main`'s layers with this branch's, so `main`'s
next PR build would miss on everything and blow its own 30-minute
timeout until the next `main` push re-warmed it. `CACHE_REGISTRY`
already exists as a knob in `hack/common-envs.mk`, so this is one env
var per job rather than new machinery. Image tags are unaffected — they
still resolve to `cozystack/<image>`, only cache refs move.
The timeout raise is not optional. The new namespace starts empty, so
this PR and the next one still build cold at ~45min, and anything that
later invalidates the cache broadly — a Dockerfile edit, a `go.mod` bump
— costs the same again. 30 minutes only ever fit a warm build, and when
it did not fit, the job reported a timeout rather than whatever actually
went wrong.
### Verification
- Both workflows are asserted to agree on the cache namespace, rather
than eyeballed.
- Confirmed the env split renders as intended: `--tag
iad.ocir.io/…/cozystack/ubuntu-container-disk:v1.35-release-1.5`
alongside `--cache-from`/`--cache-to` on
`…/cozystack-cache-release-1.5/ubuntu-container-disk:v1.35-buildcache`.
- `WRITE_CACHE` stays unset in PR builds, so they remain read-only
against the cache and never race on a manifest.
### Notes for the reviewer
- **Stacked on #3457.** Both PRs edit adjacent lines of
`pull-requests.yaml`, so keeping them independent guaranteed a conflict.
GitHub retargets this to `release-1.5` automatically once #3457 merges.
- **The filename stays `build-main.yaml`** even though it no longer
warms `main`. Renaming it would make any later backport touching
`main`'s copy arrive as a whole new file rather than a modification,
which is the failure mode the backport bot already hits on this branch.
- **One assumption I could not test locally:** the registry credentials
need repo-create permission for the new `cozystack-cache-release-1.5/*`
path. It is the same tenancy namespace that already auto-creates repos
as packages get added, so it should hold — but if it does not,
`--cache-to` fails the warmer, and the fallback is a cache-tag suffix in
`hack/common-envs.mk` instead of a separate namespace.
- **Sequencing:** the merge commit is itself a push to `release-1.5`, so
the warmer fires on merge and takes ~50-60min cold. PR builds after that
should be ~15min. The warmer cannot be triggered manually —
`workflow_dispatch` is only available for workflows present on the
default branch, and this one exists only here.
- **Out of scope, but adjacent:** `tags.yaml`, `release-e2e.yaml` and
`pull-requests-release.yaml` on this branch all still say `runs-on:
[self-hosted]`, so the release and release-e2e lanes are broken the same
way this warmer was.
### Release note
```release-note
NONE
```
Point the Build job at the namespace build-main.yaml now writes for this
branch. Without this the warmer would run and nothing would read it: the
cache-from refs would still resolve to the shared namespace main writes,
whose layers carry main's sources and miss on most of this branch's 28
packages. Builds only read the cache -- WRITE_CACHE defaults to 0, so
concurrent PR builds still never race on a cache manifest, and a miss
degrades to a cold build rather than an error.
Raise the timeout to 75 minutes at the same time. A warm build is ~15min,
but the namespace starts empty, so this PR and the next one run cold at
~45min: six ubuntu-container-disk images plus 28 packages from scratch.
Anything that later invalidates the cache broadly -- a Dockerfile edit, a
go.mod bump -- costs the same again. 30 minutes only ever fit a warm
build, and when it did not fit, the job reported a timeout rather than
whatever actually went wrong.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Nothing has ever warmed this branch's build cache. This workflow is a
copy of main's warmer that still triggers on `push: branches: [main]`, so
on release-1.5 it has never fired once -- and it targets [self-hosted],
the decommissioned cozy-runner-1 (#2937), so it could not have run even
if it did fire.
The consequence is a PR Build that runs cold end to end and exceeds its
30-minute timeout. Two separate reasons, both traceable to 39cdc8866
("bootstrap tenant workers with Talos via CABPT") landing on main six
days before this branch was cut:
* that commit dropped ubuntu-container-disk from `make image`, so main
stopped writing its six :<minor>-buildcache refs. release-1.5 is the
only branch that still builds the image, and the CI log shows all six
importers failing with "not found" -- ~28min serial, cut to ~10min by
parallelising the target in #3457.
* every other ref carries main's sources, which have diverged from this
branch, so most of the remaining 28 packages miss too -- ~38min across
the full build, with no single hot spot left to fix.
Point it at this branch, move it to the ephemeral shape main's warmer
already uses, and write to a branch-scoped CACHE_REGISTRY namespace.
The namespace matters: writing the shared refs from here would replace
main's layers with release-1.5's, so main's next PR build would miss on
everything and blow its own 30-min timeout until the next main push
re-warmed it. pull-requests.yaml has to read the same namespace, which
the following commit does.
The filename stays build-main.yaml even though it no longer warms main.
Renaming it would make any later backport touching main's copy arrive as
a whole new file rather than a modification, which is the failure mode
the backport bot already hits on this branch.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every release-1.5 PR Build job dies at the 30-minute timeout, and all of
it is spent in one target. `image-ubuntu-container-disk` builds a disk
per Kubernetes minor -- six of them -- and each spends ~4-5min inside a
libguestfs appliance installing that minor's kubelet/kubeadm into its own
copy of the cloud image. Serially that is ~28min of the 30min budget:
v1.35 6:00, v1.34 4:21, v1.33 4:27, v1.32 4:39, v1.31 4:42, v1.30 cancelled
None of it is cached, and no warmer will fix that on its own. Only
build-main.yaml sets WRITE_CACHE=1, and main dropped this image from
`make image` in 39cdc8866 six days before release-1.5 was cut, so the
:*-buildcache refs it reads have never been written -- the CI log shows
all six importers failing with "not found". release-1.5 is now the only
branch that builds this image at all.
The six builds share no per-version work, so run them concurrently
instead. A sub-make carries its own -j, keeping the root `make build`
serial, and buildkit dedups the shared guestfish and cloud-image stages
across the concurrent solves. --output-sync=target groups each version's
log under its target name; six interleaved buildx progress streams have
no per-solve identifier and are unreadable. Verified the per-version
buildx/echo/rm command set is byte-identical to the loop it replaces
across PUBLISH_VERSIONED, PUBLISH_FLOATING and WRITE_CACHE combinations.
Moving off the 4cpu/16gb shape because six concurrent libguestfs
appliances want ~9gb of RAM and a core apiece. The comment justifying
the small shape assumed a warm cache, which no longer holds here.
This also stops swallowing failures. The loop separated versions with
`;`, so make saw only the last version's exit status: a failed build for
any other version left its committed .tag file stale and the job green.
Under the sub-make a failure in any version fails the target.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The flux-instance template rendered spec.distribution.artifact
unconditionally, unlike the sibling optional fields (artifactPullSecret,
imagePullSecret, variant). The umbrella default is empty to use the
operator's embedded manifests (the air-gapped default from #964), but an
empty value renders as YAML null. In the flux-operator v0.50.0 CRD the
field is optional yet typed 'string' with pattern ^oci://.*$ and is not
nullable, so the rendered null (like an empty string) fails schema
validation and the API rejects the FluxInstance.
Guest Kubernetes clusters with the fluxcd addon enabled could therefore
never install it: the sub-release looped install/uninstall, the parent
kubernetes release stayed InProgress, and the cluster never reached Ready.
Guard the field so an empty value is omitted (the operator falls back to
its embedded manifests, preserving the air-gapped default) while a
non-empty artifact still renders. The template lives in the vendored
subchart, so the change is carried as a patch applied by 'make update'.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit da47b70e2c)
Two related issues caused the dashboard SPA to break with CORS errors
after the kc-access cookie expired:
1. KeycloakClient for dashboard had no webOrigins set, so the operator
never touched the field in Keycloak after initial creation. After
any rootUrl change the existing webOrigins value became stale and
blocked CORS from the SPA. Setting "+" tells Keycloak to derive
allowed origins from redirectUris.
2. oauth2-proxy returned 302 -> Keycloak /auth for unauthenticated
XHR/fetch on /api and /apis. The login page does not emit
Access-Control-Allow-Origin, so the SPA's fetch failed with CORS.
--api-route makes oauth2-proxy answer 401 without redirect, so the
SPA can handle the re-auth itself via top-level navigation.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
(cherry picked from commit be8f6b4739)
A leftover debug field "foo" in the <release>-urls Secret template
violates the core/v1 Secret schema. It went unnoticed under client-side
apply but breaks Helm upgrades under server-side apply with
".foo: field not declared in schema", leaving the release in
UpgradeFailed state. Drop the debug block.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
(cherry picked from commit 70a2284551)
Refresh the vendored virt-operator manifest from v1.8.2 to v1.8.4. The
release backports validation of the VMI checksum status fields as a
uint32 range, required on Kubernetes 1.36 where strict CRD numeric
format validation otherwise leaves VMIs stuck in Scheduled. Remaining
changes are bug fixes and CVE remediations; the embedded CRDs and RBAC
are unchanged, so only the operator image tag moves.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 192246e8b6)
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration
concern that rested on an outdated assumption — that an RWO Filesystem
backend-storage PVC pins the VM to its node. KubeVirt has migrated
RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on
migration it creates a fresh target state PVC and copies the small state blob,
so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine
(verified: the VM reports LiveMigratable=True and the copy-on-target PVC is
created on migration). Restore upstream's persistent firmware state for the
windows.* preferences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 59f035ff53)
The vendored seaweedfs chart ships connection pool settings only for the
mysql store, which cozystack does not use. The postgres2 store, which is
enabled, has no pool settings, so the Go sql pool keeps zero idle
connections and every filer metadata lookup opens a fresh PostgreSQL
connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3
operation this adds roughly 2 seconds of latency to every S3 request,
regardless of cluster load.
Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB
object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after
setting these values; pg_stat_activity confirmed connections were
previously recycled every ~200ms and are now reused.
max_open is capped at 40 so that two filer replicas stay below the CNPG
default max_connections of 100.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
(cherry picked from commit 49cbea8c73)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 47638f8569)
Aleksei Sviridkin