Promotes `v1.6.3-rc.2` to stable `v1.6.3` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.3` tag,
retagging the candidate and container digests to `v1.6.3` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.
⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.3-rc.2`.
ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.
✅ Includes `docs/changelogs/v1.6.3.md`, which finalize uses verbatim as
the published release body.
✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.3`), generated from the `release-1.6.3` staging
branch. **Do NOT merge that PR until `v1.6.3` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).
Address review of the version generator (hack/update-versions.sh):
- Fetch tags with `skopeo list-tags` (registry v2 /tags/list) instead of
the Docker Hub v2 API, whose anonymous pagination caps at 1000 entries
and then returns an error page with no `.results`, silently truncating
the list so the frozen 24.9.2.42 fell outside the window and the run
failed. skopeo is already in build-deps and is how the sibling
generators (postgres, redis, mongodb, opensearch, kubernetes) fetch.
- Make a frozen default tag missing from the registry a hard error
instead of silently falling back to a newer patch, which would move
the default image for existing installs.
- Restore 0644 on the regenerated files (mktemp's 0600 survived the mv).
- Exclude /hack from the chart tarball via .helmignore.
- Refresh files/versions.yaml: v25.8 -> 25.8.32.4, the true latest 25.8
patch now visible with the complete tag list.
- bats: add a missing-frozen-tag hard-error case and make the atomicity
test reach the resolution phase after a valid default rather than
failing at the default-in-set check.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit e707475f96)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
Add a `version` value (v25.8, v25.3, v24.9; default v24.9) to the
ClickHouse chart, backed by files/versions.yaml and resolved by
templates/_versions.tpl for both the ClickHouse server and Keeper
images, so the Cozystack API returns the engine version (spec.version)
like the other managed-DB charts. Default v24.9 -> 24.9.2.42 keeps
existing installations byte-for-byte identical.
The maintainer-run hack/update-versions.sh generator intersects the
server/keeper Docker Hub tags byte-collated (LC_ALL=C) so comm keeps
tags across the X.9 -> X.10 boundary; freezes the default major to the
tag it already ships (read from versions.yaml, so the freeze follows the
default); errors on a dropped default or an unresolvable major; writes
atomically; and runs on stock macOS bash 3.2 / BSD. Covered by
hack/clickhouse-update-versions_test.bats (10 cases) and the
tests/version_test.yaml helm-unittest suite.
Refs: #1246
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 3c78c0be56)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
Backport of #3552 to `release-1.6`.
Tenant cilium HelmRelease on this branch sets
`install.remediation.retries: -1` with no `strategy:`, so
helm-controller uses its default install remediation, which is
uninstall. Cilium is the tenant cluster's only CNI, so a tenant whose
install misses its budget gets the CNI torn down and reinstalled once
per cycle, and `-1` lets that repeat without end. `RetryOnFailure` keeps
applied manifests in place and retries the failed install as an upgrade
instead. Strategy is set on both actions because the retry of a failed
install runs as an upgrade.
Shipped broken in v1.6.0, v1.6.1 and v1.6.2. `RetryOnFailure` appears in
none of the 19 helmrelease templates on this branch, and the Flux CRD
release-1.6 already ships accepts it - enum at
`spec.install.strategy.name` and `spec.upgrade.strategy.name`, with the
`retryInterval` CEL guard.
Cost is carried over from the original PR: a genuine upgrade failure now
retries on failed manifests instead of rolling back. Controller can't
tell a retried install from a real upgrade, and what it replaces was an
unbounded rollback-and-retry flap.
Auto backport did not apply here, and it was not a conflict. Bot picked
four commits from the PR branch, three of them empty `ci: re-run`
commits, and `git cherry-pick` exits non-zero on an empty pick with no
unmerged paths, which the action reads as a conflict. Picking
`80a209044` alone is the whole backport and applies with a line offset
only. Resulting tree matches the bot's four-SHA-with-three-skips recipe
byte for byte.
### Testing
- `make test` in `packages/apps/kubernetes` - 18 suites, 186 tests
green.
- New suite is mutation-proven rather than revert-proven: setting
`install.strategy.name` to `Rollback` reds exactly the install test,
removing the `upgrade.strategy` block reds exactly the upgrade test,
other 185 unaffected both times.
- `make unit-tests` and `make test-controllers` green, `make generate`
leaves no drift.
- No migration touched, `targetVersion` still 54.
```release-note
fix(kubernetes): retry the tenant CNI install instead of uninstalling it
```
Backport of #3791 to release-1.6.
The automated backport stopped on a conflict in
packages/core/installer/values.yaml and committed the markers: the branch
pins the operator at v1.6.2 while the change carried main's v1.6.0 pin
alongside the new key. Resolved by keeping this branch's image and taking
only the new disableTelemetry key with its comment, which is what the
backport was for.
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Hand-backport of #3443 to `release-1.6`. The change merged to `main` on
2026-07-24, after v1.6.0 was cut, so it never reached any 1.6 patch and
is currently slated to wait for 1.7. This lands it in 1.6.2.
The cozystack-controller meters S3 bucket sizes for billing by querying
SeaweedFS metrics from a VictoriaMetrics vmselect it discovers via the
`namespace.cozystack.io/monitoring` label on the tenant namespace. That
discovery assumes the monitoring stack scraping SeaweedFS runs in the
same cluster. Some deployments host SeaweedFS (API and metrics both) in
a separate cluster, where discovery can never produce a reachable
endpoint. This adds a single controller flag,
`--seaweedfs-metrics-endpoint`, exposed as the chart value
`cozystackController.seaweedfsMetricsEndpoint`, to point bucket-size
metrics at a remote Prometheus-compatible query API. Empty keeps the
existing label-based discovery, so behaviour is unchanged by default.
### Backport adaptations
Clean cherry-pick of `95e4faf68`; no adaptations required. `go build
./cmd/...` and the full `internal/controller` test suite pass.
### Screenshots
Not applicable — no UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
### Release note
```release-note
feat(workloadmonitor): add --seaweedfs-metrics-endpoint to fetch S3 bucket size metrics from a remote Prometheus-compatible endpoint, for deployments where SeaweedFS and its monitoring stack run in a separate cluster
```
The tenant cilium HelmRelease carried no install strategy, so Flux
applied its default, RemediateOnFailure, which uninstalls the release
between attempts. Cilium is the tenant cluster's only CNI, so every
cycle takes it away and gives it back when the next install applies the
manifests again, and retries: -1 lets that repeat without end.
What sets a cycle off is an install that outruns its budget, which can
happen while the tenant still has no registered node, since the operator
Deployment cannot become Available until one exists. Remediation neither
causes that delay nor is fixed by this change; what it adds is a
repeated teardown of the CNI on top of a delay that resolves itself once
the node arrives. This removes the amplifier, not the cause.
RetryOnFailure keeps the applied manifests in place and retries a failed
install as an upgrade, so the rollout completes on its own once a worker
registers. Both actions carry the strategy because the retry of a failed
install runs as an upgrade, which would otherwise fall back to upgrade
remediation. It is the strategy and retry interval cozystack-api and
cozystack-operator already set on every HelmRelease they generate,
except that they leave remediation nil where this release keeps it. The
other tenant addons the chart renders still carry the remediation-only
shape; this one is where the teardown removes the cluster's only CNI
rather than churning a single component.
The cost is that a genuine upgrade failure no longer rolls back either,
it retries on the failed manifests instead. That is forced, since the
controller cannot tell a retried install from a real upgrade, and what
it replaces was an unbounded rollback-and-retry flap rather than a
stable safety net.
The remediation blocks stay, since they cover the branches no strategy
reaches: an absent release, which consults the install block, and an
out-of-sync one, which consults upgrade.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 80a2090444)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5bcdb8cfb8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
On the useSystemBucket / cozy-default flow the chart pointed the
clickhouse-backup sidecar's S3_ENDPOINT straight at the projected
cozy-backups-creds.endpoint, which the backup credentials projector
stores as a bare host by design. altinity/clickhouse-backup runs
S3_ENDPOINT through the aws-sdk-go-v2 endpoint resolver, which rejects a
scheme-less value with "Custom endpoint 's3.host' was not a valid URI",
so every backup and restore on the default flow failed (reproduced on a
live v1.6.2 cluster). ClickHouse was the only cozy-default app hit by
this: the CNPG/etcd/velero strategies get a scheme'd URL from the
backupstrategy-controller.endpoint helper, and mariadb/foundationdb
deliberately pair a bare host with a TLS flag.
Prepend https:// on the useSystemBucket path via a helper env plus
$(VAR) expansion (a secretKeyRef value can't be string-interpolated),
mirroring the platform's endpoint helper which force-prepends https for
the always-TLS external S3 ingress. The legacy and bring-your-own-Secret
paths keep reading endpoint verbatim, since that value is tenant-supplied
and already carries its own scheme.
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit d646c15f8d)
SSL_CERT_FILE alone does not narrow trust. crypto/x509 replaces only its file
list and then scans SSL_CERT_DIR unconditionally, and the pinned proxy image
ships /etc/ssl/certs/ca-certificates.crt, so the connection guarding the KEK
trusted the private CA plus every public root — a mis-issued certificate for the
Vault hostname was still accepted. Set SSL_CERT_DIR to the mount as well, which
collapses the pool to the CA the operator configured.
An unvalidated caBundle failed open on top of that: a non-PEM value is dropped
silently by AppendCertsFromPEM, leaving the proxy on the image's roots with no
error at all. Reject it at render time, the way backend and auth already are.
Rotating an inline caBundle had no effect either. The pod template did not
change, so no new pods, and crypto/x509 memoizes the pool per process — the
upgrade reported success while the old CA stayed in use until some later
restart. Derive a checksum/vault-ca annotation from the bundle so the rollout
happens with the change.
Also add vault.caSecretKey: the projected key was hardcoded to ca.crt, and a
bundle stored under another name (a cert-manager Certificate from an ACME issuer
emits no ca.crt at all) wedged the pod at mount time with no way to correct it
from values.
Drop the claim that the database leg verifies against its own CA pool: the chart
never sets KKP_BACKEND_CA_FILE, so that leg is plaintext today, as the note 30
lines above already says. The conclusion holds — it never consults the system
trust store — but for the other reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 517e33f67b)
The volume and mount that consume the Vault CA sit at pod level, outside the
vault-transit env block, while $vaultCASecret was seeded from caSecretName
regardless of the backend. With kms.backend=static and caSecretName set, the
Deployment mounted a Secret the chart never renders, so the proxy — the single
choke point for all Keycloak DB traffic — would sit in ContainerCreating.
Gate both CA keys, and the mutual-exclusion check between them, on
vault-transit, so the whole vault.* subtree behaves the same way under static:
ignored. Project only ca.crt from a pre-seeded Secret, which keeps a
cert-manager tls.key out of the container and turns a bundle stored under
another key into a mount failure rather than an opaque TLS error later.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 98ef5413f4)
The proxy could only reach a Vault whose certificate chains to a publicly
trusted root, so a Vault fronted by an internal load balancer with a
self-signed certificate was unreachable over HTTPS.
Allow the CA to be supplied inline (caBundle, the chart renders the Secret) or
as a pre-seeded Secret (caSecretName), mounted and exposed as SSL_CERT_FILE.
That redirects only the system trust store, which nothing but the Vault client
consults — the database leg verifies against its own CA pool.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
(cherry picked from commit 3531553176)
Backport of #3955. The automated cherry-pick carried only the first
commit of that PR and committed the values.yaml conflict, so this
resolution brings the branch to what actually merged to main: the port
comes from a single helper feeding the container port, the Service
targetPort and the policy, the rationale sits in template comments
rather than being rendered into every manifest, and the dead egress
default-deny toggle is gone.
Two deliberate differences from main. The image stays on the digest this
release line already ships rather than main's older one, and the
--client-ca-file argument is left out along with the values key it
reads: both belong to #2698, which was never backported here, and the
image on this line does not take the flag.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Four of the five findings share one root cause: a node in the tenant
tree can attach to its nearest *visible* ancestor when its real parent
is inaccessible, and per-row actions then derived their target from
that bridged ancestor.
The Edit action on a bridged tenant row pointed at a Tenant CR that
does not exist. The CR lives in the real parent's namespace, so the
target now comes from realParentNamespace — and Edit is offered only
when that namespace is readable, because otherwise there is no CR the
user could open and the button could only ever dead-end. The test that
asserted two Edit buttons had frozen the broken target as expected
behaviour; it now names the rows it expects, and the Edit button gained
a title so that assertion means something.
An inherited module chip is rendered only when the providing tenant is
NOT visible, yet it was a live button: clicking it denied the instance
GET and switched the active tenant to a namespace outside the visible
set, which the selector then bounced to an arbitrary fallback. It is
now an inert badge that states where the module comes from.
InfoRedirect's missing-definition branch hardcoded /console, throwing an
admin out of the portal its own docstring promises to preserve.
The sidebar's alsoMatch used a bare startsWith, so /admin/apps would
light up for /admin/apps-v2. It now matches whole path segments.
And SecretsTab let a rejected clipboard write escape as an unhandled
rejection, leaving the button silent — a denied permission or an
insecure context was indistinguishable from a no-op. It now reports the
failure.
Declined: CodeRabbit asked to split `import { useK8sList, type K8sResource }`
into a separate `import type`. The inline modifier is valid under
verbatimModuleSyntax and is what 26 other imports in this console use,
so the change would make the file the odd one out.
Reported-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 724354ffe1)
Opening the dashboard without a path sent the user to the catalog, and
the header gave Marketplace the primary emphasis. The Console is the
better entry point: it shows what the tenant already runs, which is
what someone opening the dashboard almost always came for. The
Marketplace stays one click away for when the intent is to add
something new.
The landing path becomes a named constant beside the portal helper, so
the redirect and the tab emphasis cannot drift apart, and the library's
own default tab set is reordered to match rather than contradict its
only consumer.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit bfd4ea0643)
Copying a secret key required revealing it on screen first, and the
button gave no sign that anything reached the clipboard. The value the
list already carries is enough to copy, so the button works while the
key stays masked, and it briefly turns into a check mark to confirm.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit cfd5444f16)
Detail pages offered a fixed tab set, so Info showed Workloads, Services
and Ingresses tabs it never fills; tabs now appear only for resource
groups the instance actually owns. The probes reuse the tabs' own list
refs and label selector, so they share one cache entry and one watch.
Back and the sidebar followed the plural instead of the entry point: a
module page sent the user to a one-item list and lit nothing in the
sidebar. Module pages now return to Modules and keep it highlighted,
while Info returns to Tenants and highlights that, matching the trees
they are reached from.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit e75500682c)
The page showed LoadBalancer services of the selected tenant only, so an
operator had to switch tenants one by one to find an address. It now
lists them for every visible TenantNamespace in a single table with a
Namespace column. A namespace whose services the user cannot list
contributes no rows instead of failing the page.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 1279652b56)
The Modules page listed the selected tenant's modules as Enabled or
Disabled cards, so a module a parent tenant provides looked disabled and
clicking it led to a Tenant CR that does not exist in the child's
namespace — an endless spinner.
It now covers every tenant the user can access, as the same hierarchy
tree the Tenants page draws, with each tenant's modules as chips
carrying the module icon, the namespace running it and its readiness.
Availability comes from the `namespace.cozystack.io/<module>` label on
each TenantNamespace, which names the providing namespace: the tenant
itself for a local module, an ancestor for an inherited one. An
inherited chip appears only when that ancestor is out of the user's
reach, since otherwise its own chip already shows the module. Clicking
any chip opens the module in whichever tenant runs it.
Tenants whose whole subtree has no modules collapse into a
"+ N tenants" branch, and the Info module is left out entirely as every
tenant carries it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 21b8a0e2f0)
The flat table becomes a tree built from TenantNamespaces, whose
`tenant.cozystack.io/<ancestor-ns>` labels carry the full ancestry. A
tenant whose parent is not accessible attaches to its nearest visible
ancestor instead of disappearing, and rows are labelled relative to that
parent, so a sub-tenant reads as `crpjxhwm` rather than
`whmcs-crpjxhwm`. Subtrees collapse from the row itself.
Each row also carries its own actions: Info opens that tenant's Info
page, and Create Tenant orders a sub-tenant at any level of the tree.
Editing a tenant switches to the namespace holding its Tenant CR — the
parent's, or its own for the hierarchy root.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 7a48d455ac)
Administration (Tenants, Modules, External IPs, per-tenant Info) leaves
the Console sidebar for Admin, which is now always visible: those pages
need no special permission. The two cluster-wide operator areas keep
their own gates, so Capacity and Backup Classes still appear only for
users who can open them.
Resource routes become portal-aware through a shared base-path helper,
so a detail, edit or order page opened from Admin stays under /admin
instead of jumping back to /console.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit ef73a0e47c)
Open the grouped application list right away instead of the
"Choose a product to deploy" page and its Show-all-apps button.
Category links keep filtering to a single category.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 5178e169b1)
Promotes `v1.6.2-rc.1` to stable `v1.6.2` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.2` tag,
retagging the candidate and container digests to `v1.6.2` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.
⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.2-rc.1`.
ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.
✅ Includes `docs/changelogs/v1.6.2.md`, which finalize uses verbatim as
the published release body.
✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.2`), generated from the `release-1.6.2` staging
branch. **Do NOT merge that PR until `v1.6.2` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).
Fixes the false `promotion changed the container repository/digest set`
that blocks #3904.
`normalized_refs()` in `hack/verify-promoted-packages.sh` gives a
host-less ref a repository equal to its own tag.
`hack/lib/image-refs.sh` emits two entries per shape-3 map, the
correctly joined ref plus shape 1's recursive scrape of the bare `tag:
<tag>@sha256:...` scalar, and on that bare scalar `repo="${image%:*}"`
is a no-op because there is no `/` and no `:` in it. Promotion rewrites
exactly those tags, so `v1.6.2-rc.1` to `v1.6.2` shows up as six changed
repositories whose digests are identical on both sides.
`promote-retag.sh` never hits this because it drops host-less refs
through its ownership filter, the verifier had no equivalent.
Host-less refs now compare on digest alone, which is the whole of the
container identity such a ref carries. Checked against the real
published artifacts: before the fix 61 entries each side differing by
those six pairs, after it 60 each side with the sets equal and 48
container digests identical.
Two things this deliberately does not do, both written as comments at
the fix. Not dropping host-less refs outright, because
`packages/system/kuberture/values.yaml` carries an `image:` map with a
`tag:` and no `repository:`, so shape 1 is the only rule that ever sees
its digest and a drop would silently stop proving it unchanged. And not
reusing `promote-retag.sh`'s `${REGISTRY}/` ownership filter, because
the verify job's `REGISTRY` names the private build registry while both
artifacts under comparison live on the public one, so that filter drops
all 48 refs, and the emptiness guard above tests the raw collection
rather than the filtered set, so the check would pass by comparing two
empty sets.
Second commit adds the fixture that was missing. The existing suite
writes a host-bearing string and exercises a passing rc to stable
rewrite, so it read as coverage while never building a shape-3 split
map, which the library itself calls the dominant shape. Reverting the
fix reddens the new case on the digest-set assertion, and replacing the
fix with an outright skip of host-less refs makes the verifier exit 0 on
a genuinely changed digest.
Once this is on `release-1.6` the check on #3904 re-runs against this
copy, so no `rc.2` re-cut is needed. `main` carries the same script and
wants the same fix.
No fixture in verify-promoted-packages_test.bats built a shape-3 split
map -- `repository:` on one key, `tag: <tag>@sha256:<digest>` on another
-- which hack/lib/image-refs.sh itself calls the dominant shape in the
tree. The existing fixture's single-string ref carries a registry host,
so its rc-to-stable rewrite normalizes through the other branch and
passes whether or not host-less refs are handled, which is why the suite
looked covered while a clean promotion could still be rejected.
Add two cases. The first promotes a split map whose tag moves from rc to
stable with the repository and digest held constant; reverting the
normalization fix reddens it on
"promotion changed the container repository/digest set", diffing
v9.9.9-rc.3@sha256:eee... against v9.9.9@sha256:eee... -- the same
tag-as-repository pair, with identical digests, that CI reported.
The second pins the property a blunter fix would break. A digest-only
`image:` map with no `repository:` (the kuberture shape) is seen by no
rule but shape 1, so it must still contribute its digest: the case moves
that digest and nothing else, and asserts the rejection comes from the
digest set rather than a neighbouring leg. Replacing the fix with an
outright `continue` over host-less refs makes the verifier exit 0 on that
changed digest and reddens the case at its exit-status assertion.
Negative assertions are counted rather than `!`-negated: under
hack/cozytest.sh a `! grep -q` cannot fail, so it would be a comment
shaped like an assertion.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages.sh reported "promotion changed the
container repository/digest set" for a v1.6.2 promotion that moved no
container bytes at all: all 48 real repo@digest refs and all 48 digests
were byte-identical between the rc artifact and the promotion candidate.
The six entries it flagged as changed repositories were not
repositories. They were bare tag strings -- v1.6.2-rc.1 on the rc side,
v1.6.2 on the candidate side -- pairing one-to-one by identical digest.
hack/lib/image-refs.sh emits two entries per shape-3 split map: the
correctly joined repository@digest, plus shape 1's recursive scrape of
the raw .tag scalar, which carries no repository. The library documents
that degeneracy and leaves host-less refs to its callers;
promote-retag.sh drops them, the verifier had only a
*/cozystack-packages exclusion. So `${image%:*}` -- a no-op on a string
holding no colon -- handed the tag over as the repository, and promotion
rewrites exactly those tags.
Set an empty repository for a ref with no `/` ahead of its digest, so it
compares on the digest, which is the whole container identity such a ref
carries and the only part promotion must not move. Dropping these refs
instead would be wrong rather than blunter: packages/system/kuberture
carries an `image:` map with a `tag:` and no `repository:`, so shape 1
is the only rule that sees its digest, and skipping it would silently
stop proving that digest unchanged.
Reproduced over the release-1.6 packages tree with the shape-3 tags
rewritten rc-to-stable: before, the six pairs above diverge; after, both
sides normalize to 60 identical entries, kuberture's digest included and
kube-ovn's real repository still visible through its shape-4 join.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
v1.6.1 was promoted with the rc e2e gate bypassed, and its promote PR
body says so verbatim. The reason is mechanical rather than anyone's
judgement call: workflows run from the ref they fire on, release-1.6's
`tags.yaml` has no `rc-e2e` job and this branch has no `e2e-tag.yaml` at
all, so when `v1.6.1-rc.1` was pushed there was nothing to produce
evidence the gate could find. Bypass was the only outcome available.
This closes that. Adds `e2e-tag.yaml` verbatim from main, which works as
written here because `hack/e2e-chainsaw/` and the `test-chainsaw` target
already exist on this branch. Splices main's `rc-e2e` job into
`tags.yaml`. Adds the `verify-release-candidate` job to
`pull-requests.yaml`, and the candidate-verification step to
`pull-requests-release.yaml` positioned before tag creation, publish and
retag. Plus `hack/verify-promoted-packages.sh`,
`hack/lib/promoted-packages.sh`, `hack/validate-changelog.sh` and two
bats suites.
All five files `promote-rc.yaml`'s preflight demands are present now
with their markers, where this branch previously satisfied one of five.
The e2e job name is kept byte-identical because promote-rc matches on
exactly that string, and `hack/promote-gate-contract.bats` now pins the
pair so the next drift gets a red test instead of a silent bypass.
One deviation worth knowing about. The verify job is keyed on author
plus a `release-` head branch, not on the presence of the `release`
label. On #3550 the PR was created at 09:35:31Z and the label arrived as
a separate `labeled` event at 09:35:32Z, and this branch's
`on.pull_request.types` has no `labeled`, so a label-keyed job would
satisfy the preflight marker and then never fire. Adding `labeled` was
the other option but on main it is interlocked with a concurrency split,
a `plan` guard complement and `e2e-fork.yaml`, so keying on the author
is the narrower change. Checked against every release-labelled bot PR
back to v1.0.7, the bot's other release PRs use `changelog-v*` heads so
they do not collide.
myasnikovdaniil