mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-29 08:07:33 +09:00
myasnikovdaniil
cc66e22952
## What's broken
`collect_refs` in both `hack/nightly-mirror.sh` and
`hack/promote-retag.sh` recognises three values shapes. The one most
package Makefiles actually write is not among them:
```yaml
image:
repository: <registry>/linstor-csi
tag: main@sha256:<digest> # digest embedded in the tag
```
Package Makefiles set `.image.tag` to `"$(IMAGE_TAG)@$(digest)"` in a
single `yq` call rather than maintaining a separate `digest` key. That
matches neither the `{repository, tag, digest}` rule (no `digest` key)
nor, usefully, the single-string rule — that one matches the bare tag
value, which carries no repository, so `ref_repo()` reduces
`main@sha256:...` to the repository `main` and the ownership filter
drops it.
Being dropped is silent. Both scripts then report success having simply
never seen those refs.
The same emit-then-drop hits a second class, found in review: refs whose
registry host does not live inside `repository` at all.
`keycloak-operator` splits it into a sibling `registry` key, and
`kubeovn` keeps it in the document-level `global.registry.address` with
a bare `repository: kubeovn`. Both are built and pushed to `$REGISTRY`
by cozystack — `kubeovn` by the `cozystack/kubeovn-chart` wrapper repo,
whose own `make image` writes all three keys — and both were emitted
host-less and discarded as third-party.
## Impact
**Nightly (`nightly-mirror.sh`) — this is the one that broke.** The
image is never copied to GHCR, *and* the unconditional host rewrite at
the end of the script still repoints its repository at the destination
registry. The result is a published reference to an image that was never
pushed there.
Confirmed against the `:nightly` tag on GHCR — the two packages using a
recognised shape are present, and eight images across six packages using
this shape are absent:
| shape | packages | `:nightly` |
|---|---|---|
| `{repository,tag,digest}` | cilium | present |
| single-string full ref | cozystack-operator | present |
| `{repository, tag:<t>@sha256}` | kamaji, kilo, linstor-csi,
piraeus-server, linstor-gui, metallb-controller, metallb-speaker,
redis-operator | **absent** |
This surfaced as the first full nightly run failing at install: the
`e2e-image-prepuller` DaemonSet never became ready (0/3, 10-minute
timeout) because `linstor-csi` and `piraeus-server` 404 on GHCR. The
prepuller only templates kubeovn/linstor/cert-manager, so it exposed two
of the eight; the other six would have failed later at HelmRelease
install.
**Promotion (`promote-retag.sh`) — same blind spot, milder
consequence.** This retags within a single registry, so the digests
already exist and digest-pinned installs still resolve — there is no
dangling reference and no 404. What is missing is the `:<version>` tag
(and `:latest` under `MOVE_LATEST=1`) for those images, so a promoted
release does not carry the tags it claims to.
The registry shows how long that has been true for the two host-split
images. `keycloak-operator` carries exactly one tag, `latest`, though
the package has shipped since 2024-11. `kubeovn`'s newest
cozystack-versioned tag is `v1.14.11-v0.38.8`, from before the 1.0
versioning switch — while `cilium`, which uses an already-recognised
shape, carries `v1.5.0` through `v1.6.0-rc.2`. Neither has received a
tag from any 1.x release.
Measured against the committed tree, the dry-run selects **19 unique
repositories before this change and 29 after** — the eight digest-in-tag
images plus `keycloak-operator` and `kubeovn`, nothing lost.
### What the nightly does *not* recover
Four of the ten are not in the root `Makefile`'s `build` target: `kilo`,
`redis-operator`, `keycloak-operator` and `kubeovn`. Their baked refs
keep the committed GHCR host, so the `SRC_REGISTRY` filter still drops
them and their `:nightly` tags remain absent after this PR. Installs are
unaffected — those digests already resolve on GHCR. The nightly recovers
the six OCIR-built images; promotion recovers all ten.
## Why this was invisible
Latent since `nightly-mirror.sh` was added — the nightly had never
completed a full run, so its mirror step was never exercised end to end.
Both test suites were green throughout. `nightly-mirror_test.bats`'s
fixture covered the single-string, `{repository,tag,digest}` and
platformSource shapes — a layout only cilium actually uses — and omitted
the dominant one. `promote-retag_test.bats` asserted "at least one
cozystack-owned image is selected", which is satisfied by the shapes
that already worked and cannot detect an omission.
Two comments actively pointed the wrong way, and are corrected here. The
header presented its list as the shapes the build writes, which reads as
exhaustive; it now says the list is empirical, names what an audit found
outside it, and says to re-run that audit when a package is added. The
ownership-filter comment named `kilo`, `kube-ovn` and
`keycloak-operator` as third parties "not pushed to `$REGISTRY`" — all
three are pushed there, and that wrong comment is part of why the
missing tags went unnoticed.
Both fixtures are extended here. The promotion test names concrete
packages rather than asserting a count or a generic pattern, since
either would drift back to proving nothing.
## Type guards
Both digest-in-tag rules check `.tag` is a string before `test()`.
Without it a non-string tag aborts the whole expression (`cannot match
with !!int, can only match strings`), and since these invocations
swallow stderr and status, that abort would silently discard **every**
ref of this shape in the same file — reintroducing this very bug in any
file that happens to carry an unquoted `tag: 1.24` next to a cozystack
image. Rule 1 is immune because its `select(tag == "!!str")` already
precedes its `test()`.
`registry`, `repository` and `digest` are guarded too, against a
narrower failure. Those keys are concatenated rather than `test()`ed,
and yq coerces int, bool, null and seq on `+` — each merely yields a ref
the ownership filter discards. Only a `!!map` aborts, raising `!!str ()
cannot be added to a !!map`, which takes the file's whole expression
with it. So that guard is insurance against a nested-map typo in a
vendored chart, not against unquoted scalars. It is a `select()`, so an
offending map drops only itself and its neighbours still resolve.
No non-string value of either kind exists in the tree today, so both are
prevention rather than an active fix.
## Testing
- Both suites pass: `nightly-mirror_test.bats` 2/2,
`promote-retag_test.bats` 4/4.
- Every new assertion was mutation-tested — removing the rule under test
turns it red. Verified individually for the shape-3 registry join, the
shape-2 registry join, the `global.registry.address` rule, and the
`repository` type guard. Also verified that a rule matching the bare tag
instead of the repository (the subtle wrong fix) fails, which is why the
assertions check the full source ref rather than only the destination.
- `promote-retag.sh --dry-run` against the real committed tree: 19 → 29
unique repositories, no losses.
- An audit of every `@sha256:` digest under `packages/*/*/values.yaml`
against the selector output accounts for all 35: 30 are selected, and
the remaining five are third-party
(`ghcr.io/lexfrei/{kuberture,ouroboros}`, `docker.io/clastix/kubectl`,
`ghcr.io/kvaps/...`) or sit outside `$REGISTRY`
(`ghcr.io/cozystack/ingress-nginx-with-protobuf-exporter/*`).
- `shellcheck` clean on both scripts at default severity and `-S style`.
## Ordering note for reviewers
Worth landing before the next release candidate is cut, rather than
treating this as routine CI cleanup.
`promote-rc.yaml` builds the release branch from the rc's staging
branch, and `pull-requests-release.yaml` runs `./hack/promote-retag.sh`
from that merged tree — so the copy of the script that executes at
promotion is the one frozen into the rc tag's tree, not whatever is on
the default branch at promotion time. An rc cut before this merges will
carry the old script and silently skip those retags when promoted.
The nightly half is also what unblocks the full-suite e2e lane, which
cannot currently publish a working install closure to GHCR.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff: it touches only `hack/nightly-mirror.sh`, `hack/promote-retag.sh`
and their two `.bats` suites. No `packages/`, no chart, no
`values.schema.json`, no `ApplicationDefinition`, no CRD or RBAC. The
`cozystack/ccp` trigger covers moving or renaming files under `hack/`
and changing what a make target does; neither applies — the anchor files
it gates on (`hack/package.mk`, `hack/common-envs.mk`) are untouched and
no make target changes behaviour.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(ci): mirror and retag every cozystack-owned image, including those whose digest is embedded in the values `tag` and those whose registry host is stored outside `repository`. Ten images across eight packages were silently skipped: the nightly published references that 404 at install, and promoted releases did not carry the `:<version>` tags they claim.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved nightly mirroring and retag promotion to recognize
digest-pinned image references across more YAML value shapes, including
digests embedded in the `tag` field and optional registry wrappers.
* Expanded support for chart-global image definitions and safer parsing
to avoid silently skipped references.
* Strengthened selection logic to exclude third-party images from copy
plans.
* **Tests**
* Expanded dry-run coverage with additional fixture cases for split
maps, `tag`-embedded digests, numeric-tag variants, and more “owned”
image assertions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.