mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
8f4c19daaeb00a2aeccf761cd96879867a4d86ce
5128
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 myasnikovdaniil
|
8f4c19daae |
chore(release): promote v1.6.1-rc.1 -> v1.6.1 (#3550)
Promotes `v1.6.1-rc.1` to stable `v1.6.1` — no rebuild. The tree pins the rc's images by digest. On merge, pull-requests-release.yaml creates the write-once `v1.6.1` tag at the merge commit, retags those digests to `v1.6.1` (+`:latest` when this is the newest stable), publishes the stable cozy-installer chart, and publishes the release — so the stable artifacts are bit-for-bit the rc that passed e2e. Do NOT squash-merge (decision B): the stable tag must attach to a real merge commit. ⚠️ **RC e2e gate bypassed** — this promotion did not verify green full e2e evidence for `v1.6.1-rc.1`. ℹ️ E2E already ran against the immutable rc, so this promote PR does not run E2E by default. A maintainer may add the `full-e2e` label to run the full suite again. ✅ Includes `docs/changelogs/v1.6.1.md`, which finalize uses verbatim as the published release body. ✅ Website docs PR opened/refreshed on `cozystack/website` (branch `update-docs-v1.6.1`), generated from the `release-1.6.1` staging branch. **Do NOT merge that PR until `v1.6.1` is published** — merging it early flips the site's latest-version pointer to an unpublished version and 404s its docs/API links (see the PR body).v1.6.1 api/apps/v1alpha1/v1.6.1 |
||
|
Myasnikov Daniil
|
8a43376ed0 |
chore(release): remove accidental tooling gitlink
Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
cozystack-ci[bot]
|
781328ebed |
docs: add changelog for v1.6.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
1a882bbfb8 |
Prepare release v1.6.1 (promoted from v1.6.1-rc.1)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
70d0c15ba7 |
Prepare release v1.6.1-rc.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
76062f585d |
[Backport release-1.6] fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix (#3542)
# Description Backport of #3510 to `release-1.6`.v1.6.1-rc.1 |
||
|
Alexey Artamonov
|
2258d2c640 |
fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.
Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
3ba087d655 |
[Backport release-1.6] chore(etcd-operator): bump etcd-operator to v0.5.4 (#3538)
# Description Backport of #3529 to `release-1.6`. |
||
|
Andrey Kolkov
|
3a7df639c9 |
test(etcd-operator): pin deployment unittest to v0.5.4
The manager image tag derives from .Chart.AppVersion, so bumping appVersion
to v0.5.4 without updating the two image literals in the deployment unittest
broke the suite (helm-unittest gates the PR via `make unit-tests`). Update
both asserts (manager container image and the agent OPERATOR_IMAGE env) to
ghcr.io/cozystack/etcd-operator:v0.5.4. Suite is green again (18/18).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit
|
||
|
Andrey Kolkov
|
ab1ec698c5 |
chore(etcd-operator): bump etcd-operator to v0.5.4
Bump the cozystack etcd-operator packages from v0.5.3 to v0.5.4. The
release is a controller bug-fix set with no API, RBAC or values changes:
- fix(controllers): derive --initial-cluster-state from phase, not from
the seed (cozystack/etcd-operator#355)
- fix(controllers): stop exempting the bootstrap seed from self-heal
(cozystack/etcd-operator#354)
- fix(controllers): extend crash-loop self-heal to memory members
(cozystack/etcd-operator#352)
- fix(controllers): switch the PDB from maxUnavailable to minAvailable
(cozystack/etcd-operator#351)
Adaptations:
- etcd-operator/Chart.yaml: appVersion v0.5.3 -> v0.5.4 (the manager image
tag defaults to .Chart.AppVersion, so this reimages the controller).
- etcd-operator/Makefile, etcd-operator-crds/Makefile: ETCD_OPERATOR_REF
v0.5.3 -> v0.5.4.
- etcd-operator-crds/templates/etcdmembers.yaml: re-vendored at v0.5.4 via
`make update`; description-only change to the /scale replicas/selector
fields tracking the PDB minAvailable fix. etcdclusters/etcdsnapshots
unchanged.
- templates/rbac.yaml left as-is: manager-role-rules.yaml is byte-identical
between v0.5.3 and v0.5.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
de8483cf05 |
[Backport release-1.6] fix(kubernetes): render the talos-reconcile Job for the default md0 group (#3536)
# Description Backport of #3535 to `release-1.6`. |
||
|
Myasnikov Daniil
|
8c4947ba44 |
fix(kubernetes): render the talos-reconcile Job for the default md0 group
The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.
For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.
Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.
The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.
Fixes #3504
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
6a5ea552e4 |
ci(release): carry the finalize fixes onto the 1.6 line (#3530)
## Why `release-1.6` was created at v1.6.0's merge commit on 2026-07-22, and the finalize fixes landed on `main` after that. Workflow files run from the ref they fire on, so a promote PR based on `release-1.6` runs **this** copy of `pull-requests-release.yaml`, not `main`'s. Shipping v1.6.1 from this line today would repeat two defects v1.6.0 already hit. ## What this carries Three upstream commits, after which this file is **byte-identical to `main`'s copy** — so later backports touching it will not conflict. | Upstream | Change | | --- | --- | | `01e1e7188` | `persist-credentials: false` on the checkout | | `f49d54a68` | publish the release with the merged changelog as its body | | `ba67fea7d` | drop `paths-ignore` from the trigger | ### persist-credentials Without it the checkout persists `GITHUB_TOKEN` as an `http.extraheader`, which silently wins over the app token injected by `git remote set-url`. The stable tag then pushes as `GITHUB_TOKEN`, which creates no workflow run, so `tags.yaml` never fires and its `generate-changelog` and `update-website-docs` backstops stay silent. That is exactly what happened to v1.6.0. ### Changelog as release body Without it the release publishes with the draft's own body, `"Promoted from vX.Y.Z-rc.N"`. **These two compound into a permanent defect on a maintenance line.** A patch's changelog is committed to `release-1.6.1`, merged into `release-1.6`, and never reaches `main` — and `update-releasenotes.yaml` only watches `main`, so nothing would ever sync it. The tag-time backstop that would have ported it is dead because of the credential bug. v1.6.1 would ship with placeholder release notes permanently, with no error anywhere. ### paths-ignore Latent rather than active. A promote PR carrying only `docs/changelogs/vX.Y.Z.md` would be dropped by the filter, producing no finalize run, no tag and no error. v1.6.1 will carry tag-string rewrites so it would not have fired, but the filter has no remaining purpose now that the promote PR always carries a changelog. ## On the cherry-pick policy `docs/release.md`'s skip rule says CI-only changes do not belong in a patch. That rule governs release *contents*; backporting release *machinery* so the line is releasable at all is a different thing, and there is precedent already on this branch — #3473, #3474 and #3514 are all CI/release backports. ## Verification `actionlint` clean. `zizmor` clean, no findings. YAML parses. No bats suite references this workflow, and `hack/promote-gate-contract.bats` does not exist on this line, so there is no stale contract test to trip. The diff against `origin/main`'s copy of the file is empty. ## Not included Three known gaps on this line are deliberately out of scope, none of which needs a code change to ship v1.6.1: - `e2e-tag.yaml` is absent and `tags.yaml` has no `rc-e2e` job, so no E2E runs on the rc. Handled by dispatching **E2E Release Tag from `main`** against the published rc tag, which also supplies the promote gate's alternate evidence. - `pull-requests.yaml` has no `labeled` trigger, so adding `full-e2e` to the promote PR starts nothing. Backporting that needs the label-event guards too, and rc-time validation makes it unnecessary here. - finalize is still the pre-#3456 monolith, so a mid-registry failure is not re-runnable and needs the documented hand recovery. ```release-note NONE ``` |
||
|
Myasnikov Daniil
|
6fe15b0bf5 |
ci(release): carry the finalize fixes onto the 1.6 line
release-1.6 was cut at v1.6.0 on 2026-07-22, and the finalize fixes landed on main afterwards, so shipping v1.6.1 from this line would repeat two defects v1.6.0 hit. Workflows run from the ref they fire on: a promote PR based on release-1.6 runs THIS file, not main's. Carries three upstream commits, after which this file is byte-identical to main, so future backports touching it will not conflict: |
||
|
myasnikovdaniil
|
090678614b |
[Backport release-1.6] fix(keycloak-configure): patch HelmRelease in release namespace on teardown (#3478)
# Description Backport of #3372 to `release-1.6`. |
||
|
myasnikovdaniil
|
3b96ef3ceb |
chore(release): don't activate kubernetes-nodes on the release-1.6 line (#3437)
## What this changes `kubernetes-nodes` is not ready to launch on the 1.6 line yet, so this removes its single include from the `iaas` platform bundle (`packages/core/platform/templates/bundles/iaas.yaml`). The platform stops activating the package while it still ships in the release; the line comes back when the feature is ready. ## Scope One line removed, nothing else. The `kubernetes-nodes` app package, its API types (`api/apps/v1alpha1/kubernetesnodes`), and the `kubernetes-nodes-rd` system package are all kept intact — this deactivates the feature, it does not delete it. ```release-note NONE ``` |
||
|
Myasnikov Daniil
|
042c69cb30 |
chore(release): don't activate kubernetes-nodes on the release-1.6 line
kubernetes-nodes isn't ready to launch on 1.6 yet. Remove only its include from the iaas platform bundle so the package still ships but stays inactive; restore the line when the feature is ready. The package, API types and RD package are kept intact. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
8d9beb4e5c |
[Backport release-1.6] fix(ci): overlay images from the PR base branch, and publish per-line artifacts (#3514)
## What this PR does
Hand backport of #3471 to `release-1.6`. Two clean cherry-picks, each
carrying its `-x` reference; the resulting tree is byte-identical to
#3471's merged state for all three files.
The overlay must be fixed **on this branch** to have any effect here.
For `pull_request` events GitHub builds the workflow from the merge ref,
and for `push` events it reads the workflow from the pushed ref — so
neither half of #3471 reaches this line while it lives only on `main`.
Concretely, `build-release.yaml` on `main` never fires for a push to
`release-1.6`, and a `release-1.6` PR keeps using this branch's copy of
`pull-requests.yaml`.
## Why the bot could not do this
The automatic backport reported success on every job and opened nothing
(run 30817682877). It is not a conflict — #3471 carries an empty commit,
`007d0b1a` (`chore(ci): re-trigger CI after a label event produced a
no-op run`, zero files). `korthout/backport-action` cherry-picks
commit-by-commit without `--allow-empty`, so that commit fails with `The
previous cherry-pick is now empty`; the `draft_commit_conflicts`
fallback then runs `git commit --all -m BACKPORT-CONFLICT`, which also
has nothing to commit, and the action aborts:
```
git cherry-pick -x
|
||
|
Myasnikov Daniil
|
e226b85f86 |
fix(ci): overlay images from the PR base branch, and publish per-line artifacts
Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.
The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.
Three deliberate choices:
* The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
per-release and rc staging branches promote-rc.yaml and tags.yaml create
(release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
images come from the tag build, and rebuilding them would be waste.
* WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
image and build-main.yaml is deliberately its only, serialized writer so
concurrent builds cannot race on the cache manifest (the 409 class #2711
fixed for image tags). A line build can overlap a main build, so writing here
would reintroduce that race. Line builds read the cache.
* A missing artifact still degrades to committed refs, but on a release line it
now emits a ::warning:: naming the branch. Silent degradation is
indistinguishable from a working overlay, which is how a mis-specified branch
filter would hide for a whole release cycle.
Cost: one `make build` per push to a maintained line, i.e. per merged backport.
hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
09f4cd10da |
fix(ci): only overlay current-main images on main-based PRs
The PR finalize job pulls cozystack-packages:main and repoints every package
the PR did not rebuild at current-main images. On a main-based PR that is the
whole point: it stops e2e from testing last-release images for everything
outside the build matrix. On a release-line PR it is wrong — the committed refs
there are not stale, they are that line`s released digests, and the charts are
written against exactly those.
So a release-line PR was installing main`s binaries onto its own charts, and the
mismatch grows with every commit main gains. #3437 is the demonstration: a
one-line change on release-1.6 that deactivates an app failed install
deterministically, twice, with
SchemaError(...core/v1alpha1.Option.spec): unknown model in reference:
"...core~1v1alpha1.OptionSpec"
from main`s cozystack-controller serving an aggregated OpenAPI that branch`s
charts cannot validate against. Nothing in the PR was broken; the lane was.
Left alone this makes every 1.6 backport look red, which is when release-branch
PRs are busiest.
Both overlay steps are now gated on `github.base_ref == main`, so a release-line
PR keeps its committed digests — the behaviour that predates the overlay.
Retargeting the overlay at a per-line artifact would be better but is not
possible today: build-main.yaml publishes only cozystack-packages:main, and the
registry carries no release-* equivalent (verified against the packages repo`s
tag list: `main` plus per-PR tags, nothing else).
hack/overlay-main-images_test.bats pins the wiring per step, so adding a third
overlay step without the guard fails the suite. Mutation-checked by removing one
guard. 12/12 green; actionlint clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
b7873117e3 |
[Backport release-1.6] fix(cozystack-basics): gate the hostname VAP policies on the VAP API (#3442)
# Description Backport of #3409 to `release-1.6`. |
||
|
Aleksei Sviridkin
|
f7e5c5fad9 |
fix(keycloak-configure): patch HelmRelease in release namespace on teardown
The pre-delete teardown Job clears the HelmRelease finalizers as its
final step, but targeted a hardcoded namespace that does not match the
namespace where the release and its RBAC live. The Job's Role and
RoleBinding are created in the release namespace and grant patch on the
HelmRelease by release name, so the ServiceAccount was Forbidden to patch
the HelmRelease in the hardcoded namespace. The Job errored and retried
forever, the HelmRelease stuck in Terminating, and the Helm release
wedged in "uninstalling", blocking any teardown or reinstall.
Target the release name and namespace so the patch matches the RBAC that
grants it. Add a helm-unittest pinning the rendered namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
96a76cb2fc |
[Backport release-1.6] fix(release): make promote-retag digest verification media-type-agnostic (#3473)
# Description Backport of #3435 to `release-1.6`. |
||
|
myasnikovdaniil
|
9b1d744d5a |
[Backport release-1.6] fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3474)
# Description Backport of #3436 to `release-1.6`. |
||
|
Myasnikov Daniil
|
a4d73b2346 |
fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls
Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.
The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.
Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.
The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.
Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.
The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.
The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.
29/29 tests green; sh -n clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
4eb4948675 |
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors
hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (#3431).
Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.
Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.
Fixes #3431
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
ba87460aae |
fix(release): prove a stable tag is absent before copying over it
Review notes from #3435, and a correction to my first attempt at them.
manifest_digest gated on skopeo`s exit status alone, so a zero exit with no
bytes hashed the empty string to sha256:e3b0c442... — a digest that looks real
and belongs to no manifest. The first fix simply required non-empty bytes, which
was worse than it looked: the caller turns an empty result into "not published,
safe to copy", so a registry answering 200 with an empty body went from an
accidental refusal (the bogus digest mismatched, so the write-once check
refused) to an actual copy over whatever the tag really held. Verified against
real skopeo 1.23: a 200 with Content-Length 0 does exit 0 with no stdout.
The helper is now explicitly three-state. A digest, or empty output for a tag the
registry PROVED absent, or non-zero for anything indeterminate — which under
set -e aborts the promotion before it writes. Absence has to be reported as
"manifest unknown"; 429, 5xx, auth and network failures all exit non-zero with
no bytes too, and reading those as "unpublished" is the same fail-open one level
down. finalize retags ~42 refs with no retry wrapper, so a single rate-limit was
enough to reach that path. skopeo`s stderr is captured instead of discarded, so
the diagnostics now name the real failure.
Tests: the write-once refusal branch the review found untested (an existing
stable tag at a different digest must be refused before any write), the empty-
body case (refuses, writes nothing, and the empty-input hash never appears in
the diagnostics), and a 429 that must not read as an absent tag. The mock now
emits realistic registry errors, since a bare non-zero exit is no longer proof
of absence. Both new guards are mutation-checked: accepting any non-zero exit as
absence fails the 429 test, dropping the empty-bytes check fails the empty-body
test.
11/11 tests green, hack/image-pin-consistency.bats still 2/2, sh -n clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
0686690381 |
fix(release): make promote-retag digest verification media-type-agnostic
skopeo inspect --format '{{.Digest}}' fails on OCI artifacts: for
cozystack-packages (config.mediaType
application/vnd.cncf.flux.config.v1+json) it prints nothing and exits
nonzero, so the v1.6.0 finalize aborted its post-copy verification right
after both copies had already succeeded, leaving 32 of 43 repos without
stable tags and skipping the cozy-installer publish. --dry-run never
executes this branch, so rehearsals could not catch it.
Compute the digest as sha256 of the raw manifest instead, which is how
registries define it and works for any media type. The helper gates the
hash on skopeo's exit status via a temp file rather than a pipeline:
a missing tag still yields an empty string (pre-check reads that as
"tag absent, proceed to copy"), never the empty-input hash
sha256:e3b0c442..., and never an abort under set -eu.
Verified against live GHCR: the raw-manifest hash of
cozystack-packages:v1.6.0 matches the digest finalize expected
(bf68208730860fa8...), and agrees byte-for-byte with the old method on
regular multi-arch images. New cozytest cases cover the OCI-artifact
digest, a post-copy mismatch, and the missing-tag path.
Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
568e1cdb25 |
fix(cozystack-basics): gate the hostname VAP policies on the VAP API
Render the route, gateway and ingress hostname ValidatingAdmissionPolicies
only where the admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy API
is served, matching the deletion-protection guard, so a cluster without that
API does not receive an unrenderable resource.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
764e30418d |
chore(release): promote v1.6.0-rc.4 -> v1.6.0 (#3425)
Promotes `v1.6.0-rc.4` to stable `v1.6.0` — no rebuild. The tree pins the rc's images by digest. On merge, pull-requests-release.yaml creates the write-once `v1.6.0` tag at the merge commit, retags those digests to `v1.6.0` (+`:latest` when this is the newest stable), publishes the stable cozy-installer chart, and publishes the release — so the stable artifacts are bit-for-bit the rc that passed e2e. The `full-e2e` label forces the full suite on this PR. Do NOT squash-merge (decision B): the stable tag must attach to a real merge commit.v1.6.0 api/apps/v1alpha1/v1.6.0 |
||
|
myasnikovdaniil
|
d3815a4ce6 |
Merge branch 'main' into release-1.6.0
Signed-off-by: myasnikovdaniil <60174387+myasnikovdaniil@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
235d38dae1 |
Prepare release v1.6.0 (promoted from v1.6.0-rc.4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
706c3f06fc |
fix(release): reuse only an OPEN promote PR, not one in any state (#3424)
## What this PR does The promote step guarded PR creation with `gh pr view <branch>`, which resolves a pull request by head branch **in any state**. A closed one satisfied the guard, so creation was skipped while the step still exited 0 — a promotion that reports success and leaves nothing to merge. Same silent-skip shape as the enumeration bug fixed in #3404: the run is green, the artifact is missing, and nothing says so. `STABLE_BRANCH` is derived from the stable version (`release-${version}`), not from the rc number, so every rc promoted to that version shares it. Once any promote PR for a version has been closed, both re-dispatch paths this workflow documents as supported are wedged: re-dispatching the same rc, and promoting a newer rc to the same version. That second one is named in the workflow's own header as the reason a leftover draft is tolerated — the tolerance was implemented for the release draft and missed for the PR. Neither path recovers by reopening the old PR, because the step's own `git checkout -B` plus force-push makes that PR's head unreachable, and GitHub then refuses: ``` 422 Validation Failed state cannot be changed. The release-1.6.0 branch was force-pushed or recreated. ``` The fix asks the question the guard meant to ask — is there an **open** PR for this head — and creates one otherwise. **Hit live.** Promoting `v1.6.0-rc.4` after #3397 had been closed: [run 29917852639](https://github.com/cozystack/cozystack/actions/runs/29917852639) went green, logged `PR already open for release-1.6.0`, and opened no PR. This currently blocks the v1.6.0 release. **Verified in both directions** against live data, so the change is not vacuous: | head branch | PR state | old guard | new guard | | --- | --- | --- | --- | | `release-1.6.0` | #3397 closed | `TRUE` → skip (bug) | `FALSE` → create ✅ | | `chore/gitignore` | #3412 open | `TRUE` → skip | `TRUE` → skip ✅ | `actionlint` exits 0 and `zizmor` reports no findings. The guard reads `${STABLE_BRANCH}` as a shell variable from the step's `env:` block rather than as a `${{ }}` expansion inside `run:`, so no expression-injection surface is added. ### Screenshots N/A — no UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, which is one file: `.github/workflows/promote-rc.yaml`. The only workflow-related trigger in the map is cozystack/ccp on "change release-prep behaviour in `.github/workflows/tags.yaml`", which this does not touch. This change also restores the documented behaviour of `promote-rc.yaml` rather than altering its contract, so nothing downstream sees a result different from what the docs already promise. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(release): promoting a release candidate now opens the promotion pull request even when an earlier promotion attempt for the same version was abandoned. Previously the workflow mistook a closed pull request for an open one, skipped creating a new one, and reported success with nothing left to merge. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved release PR creation checks so closed or previously merged pull requests no longer prevent new release PRs from being opened. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
55195ded0c |
test(e2e): capture previous container logs and stop killing the snapshot (#3346)
## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
baf0158d81 |
fix(release): reuse only an OPEN promote PR, not one in any state
The promote step guarded PR creation with `gh pr view <branch>`, which resolves a pull request by head branch in any state. A closed one satisfied the guard, so creation was skipped while the step still exited 0 — a promotion that reports success and leaves nothing to merge, the same silent-skip shape as the enumeration bug fixed in #3404. STABLE_BRANCH is derived from the stable version, so every rc promoted to that version shares it. Both re-dispatch paths this workflow documents as supported are therefore wedged once any promote PR for that version has been closed: re-dispatching the same rc, and promoting a newer rc to the same version. Neither recovers by reopening the old PR, because the step's own force-push makes its head unreachable and GitHub refuses with "state cannot be changed. The <branch> branch was force-pushed or recreated." Hit live while promoting v1.6.0-rc.4 after #3397 had been closed: the run went green and opened no PR. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Timofei Larkin
|
d5806eb1a0 |
feat(kubernetes-nodes): split worker node pool into its own app (Phase 2, additive) (#3314)
## What this PR does Phase 2 of the Kubernetes-app split ([cozystack/community#8](https://github.com/cozystack/community/pull/8)) — the **additive** half. Introduces `kubernetes-nodes`: a per-pool application that renders the worker node-pool objects (`KubevirtMachineTemplate`, `MachineDeployment`, `MachineHealthCheck`, worker `WorkloadMonitor`, and the `talos-reconcile` Job + RBAC) extracted from the monolithic `kubernetes` chart. A `KubernetesNodes` CR named `<cluster>-<pool>` attaches to a parent `Kubernetes` cluster in the same namespace by name convention (`kubernetes-<cluster>`), mirroring the `vm-instance`/`vm-disk` precedent. This PR is additive: the parent `kubernetes` chart gets exactly one scoping fix (its old-KMT preservation block is limited to KMTs owned by its own Helm release) so that attaching a `kubernetes-nodes` pool to a live cluster cannot break the parent's next upgrade. Existing clusters keep provisioning workers via `nodeGroups`. Removing `nodeGroups` from the parent and adopting existing pools into `kubernetes-nodes` releases is a follow-up **breaking** PR (Phase 2b). Key properties: - Pool objects render **byte-identical** to the parent for the same group (golden-parity check in `tests/render-parity.sh`), so the follow-up adoption never churns live worker VMs — the content-hashed `KubevirtMachineTemplate` name is preserved. - cluster-autoscaler (kept in the parent chart) discovers the pool's `MachineDeployment`s cross-release via the unchanged `cluster.x-k8s.io/cluster-name` label, namespace, and min/max annotations. - Registered as a dedicated `cozystack.kubernetes-nodes-application` PackageSource + `kubernetes-nodes-rd` ApplicationDefinition, wired into the iaas bundle (mirrors `vm-instance`/`vm-disk`). ### Downstream repositories Left unticked deliberately — this is a draft. A new user-facing `KubernetesNodes` API is added, so cozystack/website (docs) and cozystack/terraform-provider-cozystack likely need follow-ups; the trigger-map walk will be completed before this leaves draft. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: ```release-note Add the `kubernetes-nodes` application: manage the worker node pools of a managed Kubernetes cluster as independent resources, one HelmRelease per pool. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a new Kubernetes worker node pool offering with configurable sizing (CPU/memory/instance, disk), roles, storage class, GPUs, and Talos worker image settings. * Introduced a dedicated Helm package, dashboard/resource-map access, and Flux application wiring for installing/managing pools. * **Bug Fixes** * Improved safeguards against pool naming/ownership conflicts and tightened upgrade behavior for reusing previously created templates. * **Documentation** * Added full node pool configuration docs and operational notes (including supported Kubernetes versions and reconciliation expectations). * **Tests** * Added node pool rendering/validation tests plus golden parity checks between chart layouts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
284a17ba1f |
fix(release): unblock the v1.6.0 promote on the rc.4 staging tree
Promoting this branch fails in two independent ways, both landing after finalize has already created the write-once stable tag, published the GitHub release and moved :latest. chBackupClientImage had no producer and froze at v1.4.0-rc.2 while .migrations.image advanced to v1.6.0-rc.4, leaving platform-migrations pinned at two digests. promote-retag.sh copies every collected <repo>@<digest> to <repo>:<stable>, so both compete for one destination tag: the lower-sorting v1.4.0-rc.2 digest wins it and the second copy trips the write-once guard, aborting the retag with five repositories still untagged and the installer chart unpublished. Align it to the digest this branch already carries for the migrations image. Both come from the same tree, so the runner keeps the curl, jq and ca-certificates the Altinity strategy Pod needs, and that Pod overrides the ENTRYPOINT with its own command. promote-rc.yaml on main now calls hack/promote-rewrite-tags.sh, which does not exist here, so the dispatch would fail outright. Vendor it and hack/lib/image-refs.sh verbatim from main (identical blobs). That also fixes the rewrite's coverage: the old inline glob enumerated 155 files and missed 10 rc-bearing ones, which is why the earlier promote left 11 references reading v1.6.0-rc.4. The shared enumeration covers all 168 and fails the promotion if any rc reference survives. Both fixes are on main via #3405 and #3404; this carries them onto the rc.4 staging tree so v1.6.0 ships the bytes rc.4 tested. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
d0c0f399f8 |
fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404)
## The bug Promotion rewrites the rc version substring to the stable version across the vendored image references. That rewrite lived inline in `promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus `packages/apps/kubernetes/images/*.tag` **only**, on the premise that the kubernetes app was the only one whose `.tag` files carry the cozystack version. Nine other `.tag` files do, and `system/multus` seds its reference straight into a vendored upstream daemonset manifest. #3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33 of 55 first-party references still reading `v1.6.0-rc.4`**. `hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind spot from the other direction — both scanned the depth-2 `values.yaml` alone, so those images were never retagged to the stable version (**30 references selected before, 42 after**) and never mirrored to the public registry for a nightly. A retag miss leaves an image short of a tag, since the digest still resolves inside one registry. A mirror miss is worse: the host rewrite walks the same file list, so a reference that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. `v1.6.0` is the first release to go through promotion — every earlier release was a full rebuild from the tag, which stamped every file uniformly. The glob has always been wrong; the old path just made it unobservable. ## Why it could not be caught before The rewrite was workflow-inline, so there was nothing to unit test. The only way to observe the miss was to cut a release. It is a script now — `hack/promote-rewrite-tags.sh` — and `hack/promote-rewrite-tags_test.bats` round-trips it against the real tree under `make unit-tests`, with no cluster, registry or release involved. ## The fix The three call sites each carried their own idea of where a reference can live, which is how they drifted apart. That knowledge moves into `hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()` for the files, `collect_image_refs()` for the references inside them — and all three source it, so a new storage location is declared once and reaches every consumer. The rewrite also asserts its own postcondition: it scans **wider** than it rewrites and fails the promotion when a reference survives in a location the enumeration does not know about, rather than shipping it quietly. The wide scan is shape-filtered to image-reference position, because `X.Y.Z-rc.N` is an ordinary version string that other things legitimately contain — `kubevirt-csi-driver/go.sum` pins `github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:` instead would have been wrong: `system/cilium/values.yaml` keeps `tag:` and `digest:` under separate keys. `docs/agents/image-refs.md` records the contract — three tag classes, three storage shapes, the invariants, and which consumer does what. ## Not a bug: kamaji `cluster-api-control-plane-provider-kamaji` stays at `v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild of an upstream component, versioned by that component rather than by the cozystack version line, so no rewrite is owed. A test pins this so a future fix here cannot over-reach. ## Documented known gaps, deliberately not fixed here Both are pre-existing and in the **host** dimension; the version dimension this PR owns is complete, because the version always lives in a `tag` key regardless of where the host sits. - **The nightly host rewrite cannot reach a split host.** It is a literal `<src-registry>/` substring replace, so `keycloak-operator` (host in a sibling `registry:` key) and `kubeovn` (`global.registry.address`, no trailing slash) are mirrored but keep pointing at the build registry in the published tree. Fixing it needs structure-aware rewriting. - **A reference inside a gzip is invisible.** `capi-providers-cpprovider` ships `files/components.gz`, and the promote postcondition cannot catch this class at all since `grep -rIl` skips binaries. Nothing is owed today — kamaji is component-versioned and the image is still mirrored and retagged via its `.tag` — so the residual is that a nightly's kamaji ConfigMap keeps the build-registry host. ## Testing - 17 tests in the new suite; `make bats-unit-tests` green (312 tests, exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow. - Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`, `1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no false negatives. - Every test is mutation-proven. Reverting the enumeration to the old glob, deleting any of the five YAML shapes, corrupting emitted digests, reverting extras to parse-only, dropping the unreadable-file guard, broadening shape 4 beyond `global.images`, or planting an unenumerated first-party reference in any storage shape each fail a named test. ## Reviewer note, out of scope `platform-migrations` is pinned at two different digests — `core/platform/values.yaml` at `v1.5.0` and `backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned pin; no Makefile writes that key, tracked in #3143). A `promote-retag.sh` dry-run emits **two copies to the same destination tag**, so the write-once guard will fail the promotion. This is independent of this PR and needs a decision about the backup-client image version, so it is left alone here — but it blocks the 1.6.0 promote and wants fixing before the re-cut. ## Release note ```release-note Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry. ``` Fixes the promote half of #3143. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Image promotion and nightly mirroring now detect digest-pinned image references across values files, standalone tag files, and supported templates. - Stable release preparation supports RC→stable tag rewriting via a dedicated promotion script, with broader verification to ensure no RC references remain. - **Bug Fixes** - Improved completeness of ref discovery and scanning, including previously missed template/tag locations. - **Documentation** - Added and linked guidance documenting image-reference invariants and how tooling enumerates supported storage shapes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
bbfe9f9a9d |
fix(backupstrategy-controller): give chBackupClientImage a producer (#3405)
## This is a workaround `backupstrategy-controller` should not be running the platform **migrations** image. This PR does not fix that — it makes the existing reuse *safe* so the v1.6.0 promote can proceed, and documents in the tree exactly what is wrong and what should replace it. Treat it as a stopgap with a TODO, not as an endorsement of the design. ## What is actually wrong The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For that it currently pulls `platform-migrations`: | | | | --- | --- | | Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates` measures **23 MB**) | | Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate` binary, every migration script, vendored etcd CRDs | | `ENTRYPOINT` | `run-migrations.sh` | Two consequences worth naming. A tenant-adjacent backup Pod carries cluster tooling it has no use for. And because the image's own entrypoint is the migration runner, the `command:` override in the strategy template is load-bearing — drop that one line and the Pod runs platform migrations with `kubectl` and `helm` already on the PATH. It also creates the coupling that caused the bug this PR fixes: bumping `ETCD_OPERATOR_VERSION` or editing any migration script re-digests the image, forcing the backup client's pin to move for reasons that have nothing to do with backups. ## The release blocker `platform-migrations` was pinned at two different digests in the committed tree: | Location | Pin | Producer | | --- | --- | --- | | `packages/core/platform/values.yaml` `.migrations.image` | `v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` | | `backupstrategy-controller/values.yaml` `.chBackupClientImage` | `v1.4.0-rc.2@sha256:17390197…` | **none** | Promotion retags **by digest**: `hack/promote-retag.sh` copies every collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests under one repository produce two copies competing for the same destination tag — the first wins, the second hits the write-once guard, and the promotion fails. A dry run against `main` emits exactly that pair: ``` ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:17390197… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:8bf61f17… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ``` `v1.6.0` is the first release cut through promotion rather than a full rebuild. Every earlier release ran `make build`, which restamped both copies as a side effect and hid the missing producer. ## What this PR does `packages/core/platform/Makefile` now stamps both keys with the same ref, so they move in lockstep by construction rather than by instruction, and the committed value is aligned to the digest `core/platform` already carried. Moving the runner forward two minors is safe: it is used only as a curl + jq container, and the image built at `v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk add … jq ca-certificates bash curl`). ## What should replace it A dedicated `ch-backup-client` image built by **this package's own Makefile** — measured at 23 MB, 15× smaller, with none of the cluster tooling and none of the entrypoint hazard. That also deletes the cross-package stamp this PR adds, because the pin would be produced by the package that owns it. The objection that has kept this reuse in place — "a second tag release CI would have to re-digest each cut" — does not survive contact with the actual defect. Every other first-party image already pays that cost, automatically. What broke here was not an extra tag; it was **a pin with no producer**, which a package-owned image cannot reproduce. I am deliberately not doing that here: it means a new image, a new build target and a new digest introduced during a release cut. It belongs in a follow-up. ## The guard `hack/image-pin-consistency.bats` asserts the general invariant — no repository pinned at more than one digest — driven through the real promotion selector so it tracks whatever set promotion actually acts on, rather than a reimplementation that could drift from it. Plus a specific lockstep assertion so a recurrence names its cause. This check could not have been added before now: it fails on the very duplicate this PR removes. ## Testing - `hack/image-pin-consistency.bats` — both tests pass; verified they fail when the drifted pin is restored. - `make bats-unit-tests` green (294 tests, exit 0). - `make -C packages/system/backupstrategy-controller test` — 11 tests, 3 suites, green. - `helm template … --set backupStorage.bucketNameOverride=test-bucket` renders the strategy Pod with the new ref. - The Makefile's `yq` step was simulated directly to confirm the relative path and key path resolve and that comments survive the in-place edit. - Merged locally with #3404 onto `main` and re-run: the expanded ref collector in that PR surfaces no additional duplicate pins (314 tests, exit 0). ## Relationship to other work Relates to #3143 — this is one instance of its unowned-pin class, the one its discussion flagged as having no producer. The four never-built packages there remain untouched. Independent of #3404 (promote/retag/mirror storage-shape coverage); the two touch no common files and can merge in either order. Both are needed before the `v1.6.0` promote is re-cut. ## Release note ```release-note Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Ensured the platform migrations image digest is pinned consistently between the platform and backup strategy components. * Updated the backup strategy component to reference the new pinned `platform-migrations` digest. * **Tests** * Added promotion/retag consistency checks to fail on conflicting source digests for the same stable tag. * Added drift detection to ensure `platform-migrations` image references match across consumer configuration files. * **Documentation** * Expanded in-chart comments and values documentation clarifying the coupling and workaround behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
8ea589230f |
test(mariadb): add MariaDb backup/restore e2e test (#3384)
## What Reworks the never-merged [`backup-mariadb-e2e`](https://github.com/cozystack/cozystack/tree/backup-mariadb-e2e) branch onto the modern Chainsaw e2e approach ([`docs/agents/e2e-testing.md`](https://github.com/cozystack/cozystack/blob/main/docs/agents/e2e-testing.md)), rebuilt on fresh `main`. That branch bundled the whole MariaDB backup-strategy controller **plus** a `hack/e2e-apps/backup-mariadb.bats` test. The controller and the `examples/backups/mariadb/` manifests have since landed on `main` independently (`feat(platform): implement backup strategy for mariadb`), and the `hack/e2e-apps/*.bats` app suite has been replaced by `hack/e2e-chainsaw/`. So the only piece still missing was the e2e coverage — this PR delivers it the modern way. ## Changes - **`examples/backups/mariadb/` → executable harness.** Converts the demo from static numbered YAML into the same script harness the other backup demos use (`00-helpers.sh`, `run-all.sh`, `cleanup.sh`), keeping the numbered manifests as the human-readable flow that `run-all.sh` drives — so the documented flow and the automated test can never drift. `run-all.sh` provisions a `Bucket`, derives the two per-app Secrets the strategy references from `BucketInfo` + the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires an ad-hoc `BackupJob`, restores to a copy with a to-copy `RestoreJob`, and asserts the sentinel round-tripped through S3 into the copy while the source is left untouched. - **`hack/e2e-chainsaw/mariadb/` → second Chainsaw Test** (`mariadb-2-backup-roundtrip`) that drives `run-all.sh` + `cleanup.sh`, following the merged `etcd-2-backup-roundtrip` harness-driving shape: runs in `tenant-root` (the isolated e2e tenant cannot reach `tenant-root`'s seaweedfs across the Cilium egress policy), `skipDelete` since the harness owns the resources, an idempotent pre-clean step so a run killed before `finally` cannot leave a stale `Succeeded` BackupJob that false-passes, fail-fast on terminal `BackupJob`/`RestoreJob` phases, and scoped `catch` diagnostics. Cleanup also prunes the operator-side `k8s.mariadb.com` `Backup`/`Restore` CRs the Cozystack jobs own only by label. - **`hack/select-e2e.sh` → TIA rule for `examples/backups/<app>/`.** The harness lives outside the suite dir, so without this a harness-only diff produced an empty selection, which `pull-requests.yaml` turns into "skip the whole E2E stage" — a future bug-fix to `run-all.sh` would merge with its own round-trip test never run. The new rule maps `examples/backups/<app>/` to the `<app>` suite that drives it (dir name == suite name for every backup demo except `vmi` → `vminstance`), with three cases added to `hack/select-e2e_test.bats`. ## Conventions followed - No retries on deterministic steps; the single `script` step carries a justified 30m budget (2 MariaDB instances × 310s first-boot startup probe + dump/restore) and fails fast rather than polling out. - Self-contained trap-based temp-file cleanup only inside a single step; no test-level EXIT trap. - Controller-created artifacts Chainsaw cannot reclaim are pruned explicitly by `cleanup.sh`. - TIA verified: `examples/backups/mariadb/run-all.sh` now selects the `mariadb` suite (`hack/select-e2e.sh`), and editing `select-e2e.sh` itself escalates this PR to the full suite. ## Downstream Repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The change touches only e2e test material, backup examples, and CI suite-selection — no package added/renamed/removed, no `values.schema.json` / `values.yaml` / platform-values / variant / `ApplicationDefinition` / namespace / release-asset changes, and no `hack/` file moved/renamed. No downstream repository is reached. - [x] `cozystack/website` — not affected (no package or platform/docs-source change) - [x] `cozystack/terraform-provider-cozystack` — not affected (no app/schema/enum/default/prefix change) - [x] `cozystack/ansible-cozystack` — not affected (no installer values / namespace / variant / node-prereq change) - [x] `cozystack/ccp` — not affected (no `hack/` move-or-rename, no make-target behavior change) ## Notes No `make generate` needed — the change touches only `examples/`, `hack/`, and a Chainsaw suite (the edited README is under `examples/`, not a package). ## Release note ```release-note Add an end-to-end Chainsaw test for the MariaDB backup/restore round-trip, and convert the `examples/backups/mariadb/` demo into an executable `run-all.sh`/`cleanup.sh` harness. ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an end-to-end MariaDB backup and restore demonstration workflow. * Added automated backup round-trip validation, including data integrity checks. * Added reusable scripts for running the demo and cleaning up created resources. * Added automatic handling of backup credentials, endpoints, and trusted certificates. * **Documentation** * Expanded setup, configuration, execution, cleanup, and troubleshooting guidance. * Documented same-namespace restore support and cross-tenant limitations. * **Tests** * Added automated Chainsaw coverage for the MariaDB backup and restore workflow. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
efa4e41e9c |
feat(etcd-operator): bump to v0.5.3 (S3 checksum fix for non-AWS backends) (#3403)
## What this PR does Bumps the etcd-operator to **v0.5.3**, which carries the S3 request-checksum fix ([cozystack/etcd-operator#342](https://github.com/cozystack/etcd-operator/pull/342)). `git compare v0.5.2...v0.5.3` is exactly #340, #341, #342. **Why:** scheduled etcd snapshots to non-AWS S3-compatible backends (**Ceph RGW** confirmed) fail at upload. Since early 2025 `aws-sdk-go-v2` defaults `RequestChecksumCalculation` to `WhenSupported`, which stamps a CRC32 on every upload; over HTTPS a multipart part rides it as `x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`, and RGW rejects it with `400 InvalidArgument`. v0.5.3 sets `WhenRequired` on both the S3 client and the transfer manager, so multipart uploads (>5 MiB — every real etcd snapshot) carry no checksum trailer. **Changes:** - `system/etcd-operator` — `appVersion` v0.5.2 → v0.5.3 (image tag follows AppVersion; `values.yaml` keeps `tag: ""`), bump `ETCD_OPERATOR_REF`, update deployment unittest expectations. - `system/etcd-operator-crds` — re-vendor CRDs from v0.5.3. `etcdmembers` gains the additive #340 `status.version` field + **Running** printer column and picks up the `peerAutoTLS` spec field the vendored copy was already missing (all additive, backward-compatible; `etcdclusters`/`etcdsnapshots` unchanged). Fixes the stale `make update` (upstream has no `config/crd` kustomization — CRDs live in `charts/etcd-operator/crd-bases`) and makes it fail loudly on a fetch error (temp file under `set -e`, not a pipe). **Verification:** `helm unittest` 18/18 and `helm template` green on both packages; regenerated CRDs are byte-identical to upstream `crd-bases` at v0.5.3. End-to-end on a live Ceph RGW backend (freedom-portal-stage): the v0.5.2 (client-only) agent fails a multipart snapshot upload with the 400; v0.5.3 uploads an 11 MiB multipart snapshot successfully, and the operator-driven CronJob→EtcdSnapshot path reaches `Complete`. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` file-by-file against the diff (`packages/system/etcd-operator{,-crds}/**` only — a component version bump + additive CRD re-vendor). Nothing matches: not an `apps/`/`extra/` package add/rename/remove, no `core/platform` values, no variant/bundle, no platform component add/remove, no `hack/` layout or shared-tooling change, and the provider does not type the `etcd-operator.cozystack.io` CRDs (and the change is additive/opaque regardless). - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(etcd-operator): bump to v0.5.3 — etcd snapshot uploads to non-AWS S3-compatible backends (Ceph RGW, some MinIO/R2) no longer fail with `400 InvalidArgument: x-amz-content-sha256 ...`; the snapshot agent now requests a checksum only when required, on both the S3 client and the multipart transfer manager. Also brings observed EtcdMember versions (#340) and `--watch-namespace` (#341). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runtime etcd version visibility to `EtcdMember` (including a “Running” status column). * Added `peerAutoTLS` support to `EtcdMember` for operator-managed peer TLS behavior. * **Updates** * Updated the etcd-operator to version 0.5.3. * Refreshed bundled CRDs and ensured CRDs are preserved during Helm lifecycle operations. * **Tests** * Updated deployment test expectations to use the 0.5.3 operator image. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
a302bd0ddc |
docs(backupstrategy-controller): reframe the migrations-image reuse as a workaround
Running the platform migrations image as the ClickHouse backup client is not a design decision worth defending, and the in-tree comments read as though it were. Reframe them. The strategy script uses exactly curl, jq, sleep and date. For that it pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the etcd-migrate binary and every migration script into a tenant-adjacent Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the `command:` override in the strategy template is load-bearing rather than incidental. A dedicated alpine + curl + jq + ca-certificates image measures 23 MB. The standing objection to a dedicated image -- "a second tag release CI would have to re-digest each cut" -- does not survive contact with the actual defect. Every other first-party image already pays that cost automatically. What broke was a pin with NO producer, which an image owned by its own package cannot reproduce; that fix would also delete the cross-package stamp introduced here. Comment-only. No rendered output changes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
1f302d0d78 |
docs(agents): correct issue ref, mirror claim and example in image-refs
The build-list violations bullet pointed at #3143, which tracks the adjacent gap (packages in the list going stale between releases) and never covers packages absent from the list; they are now tracked in the newly filed #3416. The nightly split-host paragraph claimed both affected images are mirrored while the published tree points at the build registry — today neither ref carries the source-registry host (keycloak-operator is never rebuilt by CI, kubeovn is built externally by cozystack/kubeovn-chart), so the ownership filter skips them and the gap is latent, not live; reworded to say exactly that. The opening example paired a v1.6.0 tag with the real v1.5.0 grafana digest — a ref that will never exist — and grafana is slated to stop being built; replaced with the literal committed cozystack-api ref. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
4605bbf3db |
test(mariadb): pre-clean the backup roundtrip against stale tenant-root state
Address review feedback on #3384. The TIA rule for
examples/backups/<app>/ that an earlier revision of this commit added
has since landed on main independently (
|
||
|
Andrey KolkovandClaude Opus 4.8
|
2829f7a300 |
docs(mariadb): address review notes on backup example
- 00-bucket.yaml: correct the Secret-name comment to include the "bucket-" release prefix (bucket-<bucket>-<user>), matching the README and 00-helpers.sh. - README: note that S3_CA_NAMESPACE defaults to tenant-root independently of NAMESPACE, so a non-root tenant run must override it (and S3_CA_SECRET) too. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
d1f2822018 |
test(mariadb): add backup/restore round-trip e2e (chainsaw) via example harness
Reworks the never-merged backup-mariadb-e2e branch onto the modern Chainsaw approach (docs/agents/e2e-testing.md). The bats port that branch carried (hack/e2e-apps/backup-mariadb.bats) is dropped — the whole hack/e2e-apps/*.bats app suite has been replaced by hack/e2e-chainsaw/, and the mariadb strategy controller + example manifests it depended on have since landed on main independently. Only the e2e coverage was still missing. Convert examples/backups/mariadb/ from static numbered YAML into the same executable harness the other backup demos use (00-helpers.sh, run-all.sh, cleanup.sh), keeping the numbered manifests as the human-readable flow that run-all.sh drives — so the documented flow and the automated test can never drift. run-all.sh provisions a Bucket, derives the two per-app Secrets the strategy references from BucketInfo + the seaweedfs CA, deploys a source MariaDB with a sentinel row, fires an ad-hoc BackupJob, restores to a copy with a to-copy RestoreJob, and asserts the sentinel round-tripped through S3 into the copy while the source is left untouched. Add a second Chainsaw Test (mariadb-2-backup-roundtrip) to hack/e2e-chainsaw/mariadb/ that drives run-all.sh + cleanup.sh, mirroring postgres-2-backup-roundtrip: runs in tenant-root (the isolated e2e tenant cannot reach tenant-root's seaweedfs across the Cilium egress policy), skipDelete since the harness (not Chainsaw) owns the resources, fail-fast on terminal BackupJob/RestoreJob phases, and scoped catch diagnostics. Cleanup also prunes the operator-side k8s.mariadb.com Backup/Restore CRs the Cozystack jobs own only by label. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey KolkovandClaude Opus 4.8
|
ae946c7143 |
fix(etcd-operator): make CRD vendoring fail loudly on fetch error
Review of #3403 noted the `curl … | awk > file` recipe swallowed curl failures: the pipe's exit status is awk's, and neither the Makefile nor hack/package.mk sets pipefail, so a 404 / network error / upstream path rename on a future bump would write an empty/truncated CRD template and still exit 0. Fetch to a temp file under `set -e` (curl -f already exits non-zero on HTTP error), then awk from it — shell-agnostic (no pipefail, which dash lacks). Verified: the regenerated templates are byte-identical to the committed ones, and a bad ref now exits non-zero leaving no template files instead of silently writing empties. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io> |
||
|
Ivan Okhotnikov
|
b2e1707632 |
fix(kubernetes-nodes): drop dead ephemeralStorage guard, clarify parity scope
The ephemeralStorage guard could never fire (the pool group is built from a fixed set of chart values that never includes ephemeralStorage), so it read as protection that is not there — remove it. Note in the parity script header that only the four pool objects are compared, not the talos-reconcile Job. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
326dbe0a5b |
test(kubernetes-nodes): cover GPU and kubelet pool shapes in render parity
Extend the golden-parity matrix beyond the resources-sized pool to also diff a GPU pool and a kubelet-override pool against the parent render, so future drift in those branches (which would roll the content-hash-named KubevirtMachineTemplate) is caught in CI. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Ivan Okhotnikov
|
f21365c4f0 |
fix(kubernetes-nodes): address review nits in guard comment and NOTES
Reference the adoption migration by role (Phase 2b) instead of a number that does not exist on this branch, and correct the NOTES operational-note count. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |