PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What
Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.
## Background
This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:
- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.
## What changed
- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.
No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.
## Notes for reviewers
- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.
* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.
* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Concurrent PR builds were racing in the OCIR registry. Every package
Makefile tagged its images via `settag`, which on any non-release commit
collapsed to `:latest` — so every image across every PR was pushed to
the same tag, and parallel builds got 409s on the manifest PUT. Two
outliers (`metallb`, `redis-operator`) used hardcoded version tags and
collided across PRs unconditionally.
This PR replaces `settag` and the `TAG=latest` fallback in
`hack/common-envs.mk` with a single `image-tags` macro driven by three
env vars:
| Env | Meaning |
|---|---|
| `IMAGE_TAG` | Build-unique tag, always pushed |
| `PUBLISH_VERSIONED` | Also push `:<component-version>` (when provided)
|
| `PUBLISH_FLOATING` | Also push `:latest` |
Wired into the two workflows:
- **`pull-requests.yaml`** — `IMAGE_TAG=pr-<N>-<sha>`, both `PUBLISH_*`
off. Every PR build pushes a unique tag set; no collisions, no `:latest`
move.
- **`tags.yaml`** — `IMAGE_TAG=<ref_name>`, both `PUBLISH_*` on. Release
output preserved (component versions + `:latest`) plus a uniform
`:<git-tag>` handle on every image for traceability.
- **Local** — defaults to `IMAGE_TAG=dev`, no flags, `PUSH=0`.
All 30 package Makefiles converted to `$(call
image-tags,<repo>,<versioned-tag>)`. Image consumers reference digests
pinned into `values.yaml` by each Makefile, so the tag is decorative for
cluster-side pulls — the build-unique tag is sufficient for E2E.
Also drops the redundant `:<component-version>-<git-tag>` form (cilium,
linstor, matchbox, ubuntu-container-disk, kubevirt-*,
cluster-autoscaler). Nothing in the repo or downstream values.yaml
consumed it; the same information is preserved via the new `:<git-tag>`
plus the existing `:<component-version>` tags on release.
`ubuntu-container-disk` builds a separate image per Kubernetes version
in a loop, so its primary tag became `:<k8s-ver>-<IMAGE_TAG>` to prevent
loop iterations within a single build from pushing to the same tag.
Verified by dry-running `make -n image` across representative packages
(simple, dual-tag, skopeo, flux-push, loop) under all four contexts (PR
/ release / main / dev) — tag sets are unique and behave as expected.
### Release note
```release-note
refactor(build): PR CI now pushes images under a unique `pr-<N>-<sha>` tag per build instead of `:latest`. Concurrent PR builds no longer race in the registry. Release builds keep the existing `:<component-version>` and `:latest` tags and additionally publish a uniform `:<git-tag>` handle across every image. The redundant `:<component-version>-<git-tag>` tag (cilium, linstor, matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) is no longer published — image references in values.yaml are digest-pinned and were the only legitimate consumers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Centralized and simplified container image tagging across build
workflows.
* PR builds now use unique image tags derived from PR metadata for
better isolation.
* Added configurable flags to control publishing of versioned and
floating tags, making release tagging behavior more predictable.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2711?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Complete the strategy.backups.cozystack.io/Job driver as the generic,
app-agnostic counterpart to the Altinity driver: render the strategy
PodTemplateSpec, run it as a one-shot batch/v1.Job, poll its terminal
condition, and emit a Backup. Wire it into the existing BackupClass
dispatch via reconcileJob/reconcileJobRestore rather than a standalone
reconciler, matching the post-BackupClass backups API (the old draft
read BackupJob.Spec.StrategyRef/StorageRef, which no longer exist).
Template context follows the in-tree convention (.Application, .Release,
.Mode, .Parameters, .Backup); the JobSpec.Template doc and regenerated
CRD are updated to match. Restore re-renders the same template with
.Mode=restore; cross-namespace restore is unsupported, mirroring
Altinity.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… with others
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
align Altinity backup strategy field name for similarity with others
```
CodeRabbit (PR review on #2494) flagged that the first-party scratch
images leave the container process running as root because no USER
directive is set. Adds the standard distroless nonroot UID/GID
65532:65532 just before ENTRYPOINT in seven first-party Dockerfiles.
The chosen UID matches gcr.io/distroless/static:nonroot and the
existing convention in the kubevirt-csi-driver and kamaji images.
Statically-linked Go binaries copied with default permissions remain
world-readable, so they execute fine under the non-root identity
without an extra chown step in the builder stage.
Touched:
- packages/system/backup-controller/images/backup-controller/Dockerfile
- packages/system/backupstrategy-controller/images/backupstrategy-controller/Dockerfile
- packages/system/cozystack-api/images/cozystack-api/Dockerfile
- packages/system/dashboard/images/token-proxy/Dockerfile
- packages/system/flux-plunger/images/flux-plunger/Dockerfile
- packages/system/kubeovn-plunger/images/kubeovn-plunger/Dockerfile
- packages/system/lineage-controller-webhook/images/lineage-controller-webhook/Dockerfile
Vendored Dockerfiles (kubevirt-cloud-provider, redis-operator,
keycloak-operator, linstor-csi, multus-cni, objectstorage,
s3manager, kamaji, cluster-autoscaler) are intentionally left
untouched — their runtime user choice is upstream's call.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The root go.mod now declares go 1.26.2 (raised in the first commit
of this PR). Every Dockerfile that copies the root go.mod and builds
a cozystack-owned binary has to use a matching base image, otherwise
the in-container go toolchain has to download go 1.26 on the fly
(via GOTOOLCHAIN=auto) at every build, which slows builds and breaks
in network-restricted CI sandboxes.
Bumps:
- packages/core/installer/images/cozystack-operator
- packages/system/backup-controller/images/backup-controller
- packages/system/backupstrategy-controller/images/backupstrategy-controller
- packages/system/cozystack-api/images/cozystack-api
- packages/system/cozystack-controller/images/cozystack-controller
- packages/system/flux-plunger/images/flux-plunger
- packages/system/kubeovn-plunger/images/kubeovn-plunger
- packages/system/lineage-controller-webhook/images/lineage-controller-webhook
All eight images go from golang:1.25-alpine to golang:1.26-alpine.
Also normalizes the cozystack-operator Dockerfile FROM stage keyword
from lowercase 'as' to uppercase 'AS' while touching it.
Vendored Dockerfiles (kubevirt-cloud-provider, redis-operator,
keycloak-operator, linstor-csi, multus-cni, objectstorage,
s3manager, kamaji) are intentionally untouched: they pull upstream
sources at build time and pin their go base image to whatever the
upstream commit was tested against.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
The `backupstrategy-controller` chart declared a `Role` and
`RoleBinding` scoped to the `cozy-velero` namespace (for managing
`ResourceModifier` ConfigMaps consumed by Velero Restore). Because
`cozystack.velero` is an optional package, that namespace does not exist
in bundles that do not enable velero — and `backupstrategy-controller`
is a **default** package. Helm install aborted with:
```
namespaces "cozy-velero" not found
```
which blocked the entire
`cozy-backup-controller/backupstrategy-controller` HelmRelease on any
cluster where velero was not explicitly enabled (including the E2E
environment).
This PR moves that Role/RoleBinding into the velero chart
(`packages/system/velero/templates/backupstrategy-controller-rbac.yaml`),
so the permission grant only exists when velero is actually installed —
where it is useful. The RoleBinding subject points to the stable
`backupstrategy-controller` ServiceAccount in `cozy-backup-controller`.
### Release note
```release-note
fix(backups): moved the velero-namespaced ResourceModifier ConfigMap Role and RoleBinding from the backupstrategy-controller chart into the velero chart. This unblocks installs of backupstrategy-controller on bundles that do not enable velero (previously the HelmRelease failed with `namespaces "cozy-velero" not found`).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Reorganized RBAC configuration for the backup strategy controller by
consolidating namespace-scoped role definitions in the Velero chart
template
* Updated role bindings and permissions structure across system packages
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The backupstrategy-controller chart declared a Role/RoleBinding in the
cozy-velero namespace for ResourceModifier ConfigMap management. Because
velero is an optional package, that namespace does not exist in bundles
without velero, so Helm install aborted with "namespaces \"cozy-velero\"
not found" and blocked the default install of backupstrategy-controller.
Move the Role and RoleBinding into the velero chart so they are created
only when velero is actually installed. The RoleBinding subject points
to the backupstrategy-controller ServiceAccount in its fixed namespace
(cozy-backup-controller).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
This patch implements the Reconcile function for BackupJobs with a
Velero strategy ref.
### Release note
```release-note
[backups] Implement the Velero backup strategy controller.
```
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
This patch adds boilerplate code for a Velero strategy for the backups
API. Implementation can be added into the Reconcile function in a
subsequent PR.
###
```release-note
[backups] Add the boilerplate for a new Velero strategy for backups.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
This PR introduces a bare-bones Job backup strategy API type and stubs
out all the boilerplate for a new controller that will handle this
strategy, as well as any others in the `strategy.backups.cozystack.io`
API group.
### Release note
```release-note
[backups] Create stubs and minimal implmentations for controllers for
the strategy.backups.cozystack.io API group.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
cozystack-ci[bot]
