ci: move remaining workflows off the persistent self-hosted runner (#3268)

## What this PR does

Completes the last step of #2937 — no workflow depends on the persistent
self-hosted runner anymore, so it can be decommissioned. PR builds, the
main cache warmer, nightly, promote-rc, and pull-requests-release
already run on the ephemeral pool; this PR moves the stragglers:

- **`tags.yaml` / `prepare-release`** moves to the large ephemeral shape
(same as the main cache warmer). Release builds now warm-start from the
shared `mode=max` build cache in OCIR (written by `build-main.yaml` on
every push to main) while still pushing images to GHCR. Cache manifests
are registry-portable, and a missing cache ref degrades harmlessly into
a cold build. Without this wiring, every tag build on an ephemeral
runner would start 100% cold — the persistent runner used to warm-start
from local buildkit state.
- **`tags.yaml` / `generate-changelog` and `update-website-docs`**,
**`backport.yaml`**, **`update-releasenotes.yaml`** move to
GitHub-hosted runners: these jobs only need git and the GitHub API, no
docker or repo toolchain.
- **`pull-requests.yaml`**: the `debug` label no longer reroutes jobs to
the self-hosted runner — that path would queue forever once the runner
is gone. The label still gates the SSH breakpoint on e2e failure, which
works from ephemeral runners (the breakpoint connects out to the
rendezvous server).
- **`flux-shard-operator`** was the last image built with the stale
inline/`:latest` cache pattern (always cold since #2711); it now uses
the shared `cache-args` registry cache like every other image.

Operational notes for the actual decommission:

- The breakpoint rendezvous server (`BREAKPOINT_ENDPOINT`) is separate
infrastructure — if it happens to live on the same host as the runner,
it needs a new home before the host is retired.
- `tags.yaml` now uses the `OCIR_USER`/`OCIR_TOKEN` secrets on tag
events (already used there by `nightly.yaml` and `release-e2e` before
it).

### Screenshots

Not applicable — CI-only change.

### Release note

```release-note
ci: all CI jobs now run on ephemeral or GitHub-hosted runners; release tag builds warm-start from the shared registry build cache
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated multiple GitHub Actions workflows to use more consistent
hosted runner environments for checks, PR validation, tagging,
backports, and release note updates.
* Improved tag/release build setup for ephemeral runners, including
optional build toolchain setup and container registry authentication.
* Enhanced image build caching for the shard operator to improve
packaging efficiency.
* **Bug Fixes**
* Removed label-driven runner switching in pull request checks to make
build and test execution more predictable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

.github/workflows/backport.yaml

@@ -23,7 +23,8 @@ jobs:
         contains(github.event.pull_request.labels.*.name, 'backport-previous') ||
         (github.event.action == 'labeled' && (github.event.label.name == 'backport' || github.event.label.name == 'backport-previous'))
       )
-    runs-on: [self-hosted]
+    # GitHub API calls only — no docker, no repo toolchain: GitHub-hosted.
+    runs-on: ubuntu-latest
     permissions:
       contents: read
     outputs:
@@ -95,7 +96,8 @@ jobs:
       github.event.pull_request.merged == true &&
       github.event.pull_request.base.ref == 'main' &&
       (needs.prepare.outputs.backport_current == 'true' || needs.prepare.outputs.backport_previous == 'true')
-    runs-on: [self-hosted]
+    # git cherry-picks via backport-action — no docker, no repo toolchain: GitHub-hosted.
+    runs-on: ubuntu-latest
     permissions:
       contents: write
       pull-requests: write

.github/workflows/pull-requests.yaml

@@ -70,7 +70,7 @@ jobs:
   # required check rather than blocking e2e from starting.
   checks:
     name: Unit & controller tests
-    runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-4cpu-16gb-x86-64' }}
+    runs-on: oracle-vm-4cpu-16gb-x86-64
     timeout-minutes: 30
     permissions:
       contents: read
@@ -106,7 +106,7 @@ jobs:
       needs.plan.outputs.code == 'true'
       && needs.plan.outputs.any == 'true'
       && !contains(github.event.pull_request.labels.*.name, 'release')
-    runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-4cpu-16gb-x86-64' }}
+    runs-on: oracle-vm-4cpu-16gb-x86-64
     timeout-minutes: 30
     permissions:
       contents: read
@@ -207,7 +207,7 @@ jobs:
     if: |
       needs.plan.outputs.code == 'true'
       && !contains(github.event.pull_request.labels.*.name, 'release')
-    runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-4cpu-16gb-x86-64' }}
+    runs-on: oracle-vm-4cpu-16gb-x86-64
     timeout-minutes: 40
     permissions:
       contents: read
@@ -290,7 +290,7 @@ jobs:
       && needs.checks.result == 'success'
       && (needs.build.result == 'success' || needs.build.result == 'skipped')
       && needs.build-talos.result == 'success'
-    runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-4cpu-16gb-x86-64' }}
+    runs-on: oracle-vm-4cpu-16gb-x86-64
     timeout-minutes: 30
     permissions:
       contents: read
@@ -505,7 +505,7 @@ jobs:
     # keeps the guests' 24 vCPU / 72 GiB unchanged (no slower install) while
     # leaving 8 vCPU (25%) and 56 GiB of host headroom, which removes the
     # oversubscription that starves the probes.
-    runs-on: ${{ contains(github.event.pull_request.labels.*.name, 'debug') && 'self-hosted' || 'oracle-vm-32cpu-128gb-x86-64' }}
+    runs-on: oracle-vm-32cpu-128gb-x86-64
     timeout-minutes: 180
     permissions:
       contents: read

.github/workflows/tags.yaml

@@ -19,7 +19,11 @@ permissions:
 jobs:
   prepare-release:
     name: Prepare Release
-    runs-on: [self-hosted]
+    # Ephemeral pool, not the dedicated self-hosted runner (#2937 decommission).
+    # Serial `make build` + `make assets` is CPU-bound, so use the large shape —
+    # same as the main cache warmer (build-main.yaml).
+    runs-on: oracle-vm-24cpu-96gb-x86-64
+    timeout-minutes: 120
     outputs:
       # A release (draft or published) already exists for this tag, so the
       # build/draft/branch/PR steps are no-ops. Downstream jobs that still need
@@ -135,6 +139,14 @@ jobs:
           fetch-depth: 0
           fetch-tags:  true
 
+      # Ephemeral runners lack the flux CLI the installer's image-packages step
+      # shells out to; install if absent (idempotent).
+      - name: Set up build toolchain
+        if: steps.check_release.outputs.release_exists == 'false'
+        run: |
+          command -v flux >/dev/null \
+            || curl -fsSL https://fluxcd.io/install.sh | sudo bash
+
       - name: Login to GHCR
         if: steps.check_release.outputs.release_exists == 'false'
         uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -145,12 +157,30 @@ jobs:
         env:
           DOCKER_CONFIG: ${{ runner.temp }}/.docker
 
+      # Read-only access to the shared mode=max build cache that build-main.yaml
+      # writes to OCIR. Release builds push images to GHCR but warm-start from
+      # that cache — without it every tag build on an ephemeral runner is 100%
+      # cold (the persistent runner used to warm-start from local buildkit state).
+      - name: Login to OCIR (build cache)
+        if: steps.check_release.outputs.release_exists == 'false'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          username: ${{ secrets.OCIR_USER }}
+          password: ${{ secrets.OCIR_TOKEN }}
+          registry: iad.ocir.io
+        env:
+          DOCKER_CONFIG: ${{ runner.temp }}/.docker
+
       # Build project artifacts
       - name: Build
         if: steps.check_release.outputs.release_exists == 'false'
         run: make build
         env:
           DOCKER_CONFIG: ${{ runner.temp }}/.docker
+          # Cache manifests are registry-portable (oci-mediatypes=true) — read
+          # them from OCIR while pushing images to GHCR. A missing :buildcache
+          # ref 404s harmlessly into a cold build.
+          CACHE_REGISTRY: iad.ocir.io/idyksih5sir9/cozystack
           # Push the release tag as :IMAGE_TAG (vX.Y.Z-rc.N) and the component
           # versions as :<component-ver>. Deliberately NO PUBLISH_FLOATING: this
           # job only ever runs for a prerelease (a stable tag is rejected or
@@ -283,7 +313,8 @@ jobs:
 
   generate-changelog:
     name: Generate Changelog
-    runs-on: [self-hosted]
+    # git + node + copilot CLI only — no docker, no repo toolchain: GitHub-hosted.
+    runs-on: ubuntu-latest
     needs: [prepare-release]
     permissions:
       contents: write
@@ -492,7 +523,8 @@ jobs:
 
   update-website-docs:
     name: Update Website Docs
-    runs-on: [self-hosted]
+    # bash + git + yq + gh against the website repo — no docker: GitHub-hosted.
+    runs-on: ubuntu-latest
     needs: [generate-changelog, prepare-release]
     # generate-changelog is non-blocking — run as long as prepare-release succeeded.
     # `always()` is needed so this job runs even if generate-changelog failed/skipped.

.github/workflows/update-releasenotes.yaml

@@ -27,7 +27,8 @@ permissions:
 jobs:
   update-releasenotes:
     name: Update Release Notes
-    runs-on: [self-hosted]
+    # GitHub API calls only — no docker, no repo toolchain: GitHub-hosted.
+    runs-on: ubuntu-latest
     permissions:
       contents: write
 

packages/system/flux-shard-operator/Makefile

@@ -14,8 +14,7 @@ test:
 image-flux-shard-operator:
 	docker buildx build -f images/flux-shard-operator/Dockerfile ../../.. \
 		$(call image-tags,flux-shard-operator,v$(COZYSTACK_VERSION)) \
-		--cache-from type=registry,ref=$(REGISTRY)/flux-shard-operator:latest \
-		--cache-to type=inline \
+		$(call cache-args,flux-shard-operator) \
 		--metadata-file images/flux-shard-operator.json \
 		$(BUILDX_ARGS)
 	IMAGE="$(REGISTRY)/flux-shard-operator:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/flux-shard-operator.json -o json -r)" \