mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-02 18:16:53 +09:00
v1.6.4
699
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Yan Bondarenko
|
ead8bb3a57 |
test(kafka): wait for ZooKeeper PVC deletion
Negative PVC assertions can stop on an API read error before the deletion timeout expires. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
myasnikovdaniil
|
0dc238a94a |
[Backport release-1.6] fix(kubeovn): retry VM migration setup after scheduling (#4328)
# Description Backport of #4253 to `release-1.6`, together with #3460 which it depends on. On 1.6 kube-ovn chart and image still come from cozystack/kubeovn-chart, so draft Dockerfile had nothing to build it and kubeovn build failed (no `image` target). #3460 moves build in-tree on same kube-ovn 1.15.10 this branch runs, and brings nightly-mirror fix for kube-ovn split registry host that becomes needed once kube-ovn is built by CI. Bot draft also dropped second commit of #4253 (`fix-vmim-target-pod-retry.diff`), it is back now. I built image locally, all three patches apply to kube-ovn v1.15.10 and `TestVMIMigrationSchedulingRetry` passes. |
||
|
Myasnikov Daniil
|
316dcef772 |
test(cluster-api): drop the OCI export checks release-1.6 cannot satisfy
The CAPK build suite came over from main asserting that an image build under OCI_EXPORT_DIR writes an OCI archive with --push=0. That export path lives in hack/common-envs.mk on main and was never brought to this branch, where every build pushes, so the suite failed here on every run. What the suite is for still holds on this branch: the build stamps a digest-pinned ref into the .tag file that the image-ref tooling finds, it leaves the shipped components untouched, and a failed build or a missing digest keeps the previous pin. Those checks stay; the OCI parameter and the three archive checks go. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Yan Bondarenko
|
b3dd13b173 |
test(cluster-api): declare the subshell EXIT traps in the CAPK bats suite
hack/bats-no-exit-trap.bats is an exact ratchet, and the new suite installs
three traps inside explicit subshells without declaring them, so
make unit-tests fails on the merge commit. The header now declares the
count and says why the three are not debt.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Yan Bondarenko
|
a0e1c8a63a |
fix(cluster-api): backport live migration before host eviction
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Yan Bondarenko
|
0a359e5f4d |
fix(kubeovn): retry VM migration setup after scheduling
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Andrei Kvapil
|
cb67da1016 |
fix(nightly): rewrite an image host that is the whole scalar value
The nightly host rewrite is a literal "<src-registry>/" substring
replace, so it only reaches a host sitting contiguously in front of the
repository. kubeovn keeps its host in global.registry.address with the
bare repository in a sibling key: no trailing slash, no match. The
image was mirrored to the public registry and the tree kept pointing at
the private build registry.
The gap was dormant while kubeovn's image came from a separate
repository, because the baked ref carried the public host and the
ownership filter skipped it. Building the image in-tree stamps the
build-registry host into that key and turns it live, so fix it here.
Add an end-of-line-anchored expression that fires only when the source
registry is the complete value; a contiguous ref continues past the
host, so the two expressions cannot both fire on one reference.
keycloak-operator splits at a different boundary and stays unfixed --
neither of its keys holds the source registry as a whole. Documented,
and dormant while no CI path rebuilds it.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit
|
||
|
IvanHunters
|
36f2963883 |
test(kafka): anchor the release selector the hook suites assert on
Both suites matched "strimzi.io/cluster=kafka-test" as a bare substring,
and that is a prefix of the sibling release kafka-test2 and of
kafka-test-topics. Rendering the hook with either selector left both
suites green, so neither pinned the scoping they exist to pin. Anchor on
the leading "-l " and on the trailing delimiter; both variants now red in
both suites, and the six earlier hook mutations still red exactly where
they did.
Drop the notMatchRegex on a leading slash. It cannot fail on its own: the
anchored image pattern above rejects a leading slash first, and an empty
_cluster.images-registry renders no slash to begin with. Its rationale
moves into that pattern's comment.
Record why activeDeadlineSeconds and the 180s wait have to move together.
A deadline that fires mid-wait kills the Job, and with backoffLimit: 0 the
hook then returns non-zero, which is the wedged uninstall the deliberate
exit 0 on the timeout branch exists to avoid.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit
|
||
|
IvanHunters
|
a1430c9f2d |
test(kafka): drive the hook suite through the runner CI actually uses
The suite used bats' run helper. make bats-unit-tests does not invoke the bats
binary: it runs hack/cozytest.sh over every hack/*.bats, and that runner
implements @test and little else, so the file died at 'run: command not found'
and took make unit-tests with it. Under bats it was 5/5, which is where the 5/5
in the previous commit came from. Save $? by hand instead, the shape
app-cleanup-hooks_test.bats already uses; verified under both runners.
The fake kubectl no longer defaults DELETE_RC to 0. An unset value made every
failure-path assertion vacuous; now it exits 64 and says so. Cover both
RESTMapper wordings for an absent CRD rather than just the first.
Widen the image pattern to v[0-9]+\.[0-9]+(\.[0-9]+)? like harbor and mariadb:
pinning v1\.32 reintroduced the breakage the digest was made pattern-based to
avoid, one Renovate bump later. Assert the _cluster.images-registry override
and the no-leading-slash default, neither of which was covered: the hook is the
one image in this chart whose pull failure surfaces at uninstall.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit
|
||
|
IvanHunters
|
30dc75667e |
test(kafka): run the pre-delete hook against a fake kubectl
The hook's exit code is the whole contract: a non-zero one returns from
Uninstall.Run before any release resource goes, keeping the topic operator
alive; a zero one lets Helm remove it. delete_test.yaml matches manifest text,
so it stayed green when the timeout branch was put back to exit 1, when the
CRD-absent branch was, and when the selector was widened past this release.
Add a bats suite in the shape hack/app-cleanup-hooks_test.bats already uses for
qdrant and harbor: render the script, run it against a fake kubectl, pin the
exit code of each branch. Each of the three mutations above now turns it red.
Drop hook-succeeded from the delete policy and reclaim the Job with
ttlSecondsAfterFinished instead. The script exits 0 on the degraded path too,
where topics still hold the finalizer, so deleting on success removed the only
record of that seconds after it was written. Assert both in delete_test.yaml.
Pin the hook image by shape rather than by literal digest: Renovate rewrites
digests under packages/*/templates/hooks/ and nothing rewrites tests/, so the
literal turns the next bump red for no defect.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit
|
||
|
Yan Bondarenko
|
2a71e5c678 |
test(kafka): assert release topics exist before the deletion check
The post-delete absence assertions could pass vacuously if the two
release topics were never created. Asserting both topics first makes
the E2E prove that the pre-delete cleanup actually removed them.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Yan Bondarenko
|
30afb9a26a |
fix(kafka): delete release topics before the topic operator is removed
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
bcdb82bf0c |
test(e2e): request immediate binding for container-lane VM disks
The vminstance suite waits for each VMDisk's DataVolume to be Ready before anything consumes it. On `replicated` that works because the class binds Immediate; on the container lane's `local` class, which is WaitForFirstConsumer, CDI starts no importer for a disk nothing has consumed yet, so the DataVolume sits in PendingPopulation and the suite times out with no error anywhere. main makes the vm-disk chart request immediate binding for every source; the chart on this branch requests it only for uploads, and the lane must not change what it ships. So on the container lane, right after each VMDisk is applied, the suite waits up to 180s for CDI to create the DataVolume and its claim and annotates both with cdi.kubevirt.io/storage.bind.immediate.requested, the same request the chart already writes for uploads. Both, because the DataVolume decides what phase CDI reports and the claim is what the populator reads to start the import without a consumer. The QEMU lane annotates nothing. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
Myasnikov Daniil
|
38b1c85eb1 |
test(e2e): raise the CDI worker memory ceiling on the container lane
CDI's own 600M worker memory limit was seen to OOM the decompress and convert of a tenant worker disk near 100% on the container lane, after which CDI retries from scratch and the DataVolume cycles forever rather than failing. The same import completes at 600M on QEMU nodes, so what needs the headroom is the substrate. main passes the raise through a kubevirt-cdi chart value that this branch does not have. The last install test now merges the same figures into the CDI CR on the container lane: 4Gi and 256Mi for memory, CPU left at CDI's 750m and 100m because the failure was memory. It has to be the CR, not CDIConfig, because the operator reconciles CDIConfig.spec from the CR and reverts a direct edit, and the test waits for CDIConfig's published default to read 4Gi, since that is what CDI hands its worker pods. The QEMU lane keeps CDI's defaults. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
Myasnikov Daniil
|
0bc0beee8c |
test(e2e): constrain container-lane CSI provisioning to the chosen node
On the container lane every volume is on `local`, which is node-pinned and WaitForFirstConsumer. Without --strict-topology external-provisioner passes every topology segment as requisite and only prefers the node the scheduler picked, so linstor-csi may provision elsewhere when that node is full. A CDI importer mounts two such volumes and the scratch one is created only after the pod is scheduled, so it can land on another node and leave the importer unschedulable forever with nothing erroring. main turns the flag on in the linstor chart; the chart on this branch cannot, and the lane must not change what it ships. So the last install test, on the container lane only, adds the same csi-provisioner override to the live LinstorCluster, waits for piraeus-operator to carry it into linstor-csi-controller and for that rollout to finish, and asserts the flag on the Deployment. `args` has no patchMergeKey, so the list is restated in full as piraeus-operator v2.10.2 renders it. It runs last because the earlier tests that change platform values upgrade linstor, and an upgrade re-renders the LinstorCluster without the override. For the same reason the tenant Kubernetes suites and both vminstance Tests re-check the flag before they import, so an override lost to a later upgrade fails by name rather than as an import stuck at ImportScheduled. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
Myasnikov Daniil
|
975c26b3a5 |
test(e2e): drop the DRBD sidecar from container-lane satellites
The linstor chart adds a drbd-logger sidecar to every satellite, and on the container lane's shared kernel there is no DRBD for it to tail, so it exits at once. The satellite pod is then never Ready, piraeus never registers the node, LINSTOR reports no nodes and no PVC binds. main drops the sidecar through a chart value; the chart on this branch has none, and the lane must not change what the chart ships. So post-install prep applies a LinstorSatelliteConfiguration of its own on the container lane, ahead of the linstor waits, with the same DaemonSet patch main renders: delete drbd-logger from the generated satellite DaemonSet. The two DRBD-only initContainers are already gone through the chart's Talos satellite configuration. Once all three nodes are Online, prep reads the live DaemonSets back and fails if a satellite still carries the sidecar or none was found at all, so a configuration that silently stopped applying cannot pass for one that worked. The QEMU lane applies nothing and keeps the chart as shipped. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
Myasnikov Daniil
|
df5bbd1aea |
fix(e2e): make the OIDC teardown assertions fail closed
The three checks that System-mode artifacts are gone after the mode
switch were `if kubectl get ... >/dev/null 2>&1`, which reads every
non-zero exit as "the object is gone". An RBAC denial, a timeout or a
missing CRD produces the same exit, and stderr went to /dev/null, so the
suite reported a clean System teardown for never having observed one.
--ignore-not-found is what separates absent from unaskable: absent is
exit 0 with empty output, everything else stays non-zero. Preferred over
matching NotFound in the message, which depends on wording and, under a
runner with tracing on, captures the trace instead of the error.
Each probe is pinned individually rather than as a group: the new case
breaks one query per iteration while the other two answer absent, so
reverting any single probe to the bare form reddens the suite. Verified
by doing exactly that three times. A test that broke all three at once
would have passed on any one surviving probe, which is how the first
attempt at this test fooled me.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
3aca80a652 |
test(e2e): keep the chainsaw ops above the deadlines they contain
.chainsaw.yaml states the rule and says why: an op that expires before
its inner bounds do is a SIGKILL on the process group, which takes out
the diagnostics the failure branch exists to print, while an inner
timeout leaves a partial capture and lets the remaining steps run.
refresh-and-assert-full-health, added on this branch, breaks it. The
poll deadline is 270s, the last in-loop read can start just under it and
run 25s, and the failure branch then reads again for another 25s: 320s
of inner bounds under a 5m op. Raised to 6m.
verify-processes-and-security-context breaks it harder and did so
before this branch: 240s for the process counts, then 120s for
connectionString and 120s for generations.reconciled, sequentially, is
480s under the same 5m. Fixed here rather than filed, because it is the
same defect one step away, found by applying the rule to the neighbour.
Raised to 9m.
The kubernetes-latest ceiling gets the measurement instead of a change.
The OIDC lifecycle folded onto that cluster adds up to 1m + 600s + 1m of
passing-path waits that the 67m op does not price, and raising the op is
not available: a guard holds COZY_OP_CEILING equal to both suites'
timeout, and two suites at 79m plus their teardowns exceed the job's
215-minute cap. On the green run of this branch the whole Test took
569.72s against the 4020s op, with kubernetes-previous at 470.19s. The
ceilings only bind once something is already hung, which is what the
existing residual note (#3666) says; now it says it with a number.
Adapted for release-1.6: the kubernetes-latest op here is 40m and no
guard ties it to another figure, so its note states what it does not
price instead of citing the COZY_OP_CEILING guard, the 67m ceiling and
the main-branch measurement, none of which exist on this branch.
Assisted-By: LLM
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
18c5308a2b |
test(e2e): pin the stability-window reset and keep the fan-out test portable
The readiness gate resets its stability window when a list read fails,
and nothing held that: the existing "transient list error" case fails on
the first read, when there is no window to invalidate, so removing the
reset left the suite green. The new case breaks a read mid-window, where
a gate that kept counting through the gap would accept a release set it
never observed stable across it.
The parallel fan-out contract asked make for --output-sync, which arrived
in GNU Make 4.0 while macOS still ships 3.81 -- so on a contributor's
machine the suite died on an unknown option instead of reporting the
contract. Guarded by version, with the reason in place. CI runs 4.x, so
the contract is still enforced where it gates a merge.
Adapted for release-1.6: the --output-sync guard is dropped together
with hack/unit-test-parallelism.bats, which this branch does not have
because the unit-test fan-out it covers is not backported.
Assisted-By: LLM
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
13ebb5f94a |
test(e2e): make the HelmRelease readiness gate hold its own ceiling
Two ways the gate did not do what its arguments promise.
The list read carried no bound. kubectl defaults to no request timeout,
and the read sits above the deadline test at the bottom of the loop, so
against a wedged apiserver it blocks there and timeout_seconds stops
being a ceiling: what arrives is the job timeout as a kill, instead of
the not_ready dump this function exists to print. Bounded with kubectl's
own --request-timeout rather than a `timeout` wrapper, because the unit
suite substitutes a kubectl shell function and `timeout` execs a binary.
The final diagnostic read is bounded the same way.
`now` was sampled before the read, so a read costing more than the poll
interval was priced at zero against the deadline and bought the loop
another whole iteration past it. Sampled after, on both branches.
And the count >= minimum_count half of the gate was pinned by nothing:
every fixture returned 11 or 12 items against a minimum of 11, so the
condition could be deleted outright and all five tests stayed green.
That threshold is the direct replacement for the `wc -l` existence
backstop this branch removes from hack/e2e-install-cozystack.bats -- the
only thing separating "everything is Ready" from "almost nothing got
created", since three HelmReleases that exist are all Ready. Both new
cases were checked by reverting the code they cover.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
f75153ac0a |
test(e2e): run the merge-gating lanes on Talos containers
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Adapted for release-1.6: this branch has no e2e-fork.yaml, no soft-red
gate, no ghcr.io mirror and no backup e2e, so those hunks are dropped
together with the tests for helpers that exist only on main; e2e-tag.yaml
stays on QEMU. The platform Package generator carries the API endpoint
and nothing else, because the linstor drbd.enabled and kubevirt-cdi
importerResources overrides need chart values this branch does not ship;
the container lane gets e2e-side substitutes for them in later commits.
run-kubernetes.sh takes the storage-class helper, the local-lane pool
baseline, the scoped drain and the OIDC fold, while cleanup keeps its
best-effort `|| true`. The OIDC suite references in select-install.sh and
the chainsaw README go with the suites here rather than in a later commit.
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
a44d7982f5 |
fix(e2e): restore the default StorageClass when the fallback render fails
verify_storageclass_fallback_default deletes the `replicated` class,
renders with --dry-run=server, and restores it inline "before any
assertion exit" -- the comment's words. Under the caller's `set -eu` the
render is a plain assignment, so a failed render exits the function
there: before rc is read, and before the restore runs. The management
cluster is then left with no default StorageClass for every suite after
it, which is the opposite of what the comment promises.
Captured with `|| rc=$?`. Pre-existing shape, but the restore block is
edited on this branch anyway, and the QEMU lanes are where it bites.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit
|
||
|
myasnikovdaniil
|
78f1d2a10f |
[Backport release-1.6] fix(e2e): budget tenant scheduling separately from workload readiness (#4435)
Backport of #3579 to `release-1.6`. Tenant backend Deployment in the kubernetes suites still has one 300s budget for scheduling and image pull on this branch. Right now those suites on 1.6 mostly fail earlier, on node-join, so this starts to matter once #3455 widens that window. Cherry-picked clean with `-x`. ### Testing - `make unit-tests` green, POSIX sh sweep clean. ```release-note NONE ``` |
||
|
myasnikovdaniil
|
e037707897 |
[Backport release-1.6] fix(e2e): budget each linstor wait separately in post-install-prep (#4434)
Backport of #3715 to `release-1.6`. Both linstor waits in `hack/e2e-post-install-prep.sh` still run off one 15m deadline on this branch, so a slow linstor HelmRelease eats the Deployment's budget and install reports the wrong link as broken. Each wait now gets its own budget. Cherry-picked clean with `-x`. ### Testing - `make unit-tests` green, POSIX sh sweep clean. ```release-note NONE ``` |
||
|
myasnikovdaniil
|
4e648f046f |
[Backport release-1.6] fix(e2e): stop the sandbox cluster CIDR overlapping the pod range (#4432)
Backport of #3750 to `release-1.6`. Sandbox Talos config and platform install on this branch still both use `10.244.0.0/16`, so kube-ovn and cilium allocate from one range and cilium refuses pod sandboxes with `IP ... is already in use`. It hit `mariadb-2-backup-roundtrip` on the E2E run of #4328 here. Cherry-picked clean with `-x`. ### Testing - `make unit-tests` green, POSIX sh sweep clean. - Real check is E2E on this PR. ```release-note NONE ``` |
||
|
Aleksei Sviridkin
|
9db2357d6e |
fix(e2e): budget tenant scheduling separately from workload readiness
The tenant backend Deployment had a single 300s budget to reach
condition=Available, and two unrelated variable costs shared it. The
first is scheduling. What the suite establishes before that point is two
tenant nodes Ready, which is weaker than schedulable: a Ready node still
carries node.cilium.io/agent-not-ready until the tenant cilium agent
claims it, and a node the bringup has not finished with is
SchedulingDisabled. Scheduling took 2m18s and 1m57s in the two tenant
suites of one run, leaving the image pull to finish inside what was
left. It did not, and both suites failed with the same message for two
different shortfalls.
Wait for a node that actually accepts a toleration-free Pod before
creating the workload, on its own budget and its own failure message.
The gate encodes the scheduler's rule for such a Pod (Ready, not
unschedulable, no NoSchedule or NoExecute taint) over a custom-columns
probe, prints the node table on both outcomes so a timeout names the
taint that held it, and treats a failed probe as not-schedulable so an
API blip cannot release it. The readiness wait keeps its 300s, now
starting from a schedulable node, and dumps deployment, pod and event
state when it runs out.
Pin the workload image by digest. The tenant workers reach no registry
mirror, so nginx is pulled from Docker Hub on every run either way; the
digest fixes what that pull returns instead of leaving a floating tag
free to change size and content under a fixed deadline.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
d8e8e661fc |
fix(e2e): budget each linstor wait separately in post-install-prep
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.
Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.
Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.
It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.
A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.
The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.
Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.
The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.
The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit
|
||
|
Aleksei Sviridkin
|
66c9ac1a50 |
fix(e2e): stop the sandbox cluster CIDR overlapping the pod range
The sandbox Talos config and the platform install step both used
10.244.0.0/16, one as the Kubernetes cluster CIDR and the other as the
kube-ovn POD_CIDR, so two allocators drew from one range.
kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium is
chained behind it and allocates no pod addresses, but it still carves
its own per-node router IP and Ingress IP out of Node.spec.podCIDR,
which kube-controller-manager slices from the cluster CIDR. Sooner or
later kube-ovn hands a pod an address cilium already holds. When that
address is the Ingress IP the agent refuses the sandbox with "IP
ipv4:<addr> is already in use", and cilium never relents on it, so the
pod only proceeds if something reschedules it onto a different address.
In the failure this comes from, the same address was refused seven
times across six pod UIDs before the suite ran out of budget. When it
is the router IP nothing complains at all and only host-to-pod traffic
breaks.
Move the cluster CIDR to 100.65.0.0/16, which no other range in the
tree uses. That deliberately stops the sandbox matching the layout the
install documentation prescribes, so it is not a value to put back.
Pod addresses are unaffected: they still come from kube-ovn out of
10.244.0.0/16, and cilium does not require a chained endpoint's address
to sit inside its own allocation CIDR.
Excluding the addresses on the kube-ovn side would not work, because
cilium picks its offsets by random scan over the node's range.
The new guard checks the invariant rather than the literals, so either
value may move as long as they stay disjoint. It covers the service
and join ranges too, since landing on one of those instead is the next
way to get this wrong.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
(cherry picked from commit
|
||
|
Andrei Kvapil
|
f3fff8b1d1 |
fix(e2e): widen tenant node-join deadline to 18m
The kubernetes-latest and kubernetes-previous suites flaked on the
worker node-join wait under host load. In run 30260770694 both failed at
exactly 12m with zero tenant Nodes registered and the tenant cilium HR
still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed
the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m
into this wait and the guests then had ~10m to import Talos, boot,
register a kubelet and let cilium turn the nodes Ready.
This single deadline absorbs the entire worker bring-up (the
machinedeployment replicas=2 gate clears before the VMIs even exist), so
under storage pressure 12m is too tight. A less-loaded fleet run passed
both suites unchanged, confirming load-induced slowness rather than a
stuck bring-up. Widen the deadline to 18m, which restores margin and
still sits well inside the 40m chainsaw step timeout.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit
|
||
|
Andrei Kvapil
|
b4c06a0fbe |
fix(e2e): disable guest fsync on ephemeral CI VM disks
The e2e QEMU guests boot with three raw virtio -drive lines that carry
no cache= option, so QEMU defaults to cache=writeback and honors every
guest fsync. The management-cluster Talos etcd fsyncs on each Raft
commit, while containerd unpacks dozens of images and CDI imports
multi-GB VM disks onto the same host-backed raw files during the
VM-heavy suites. The fsync storm serializes etcd's writes into
multi-second stalls, the mgmt apiserver times out lease PUTs, and many
operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji)
lose leadership at once — failing whichever chainsaw suite is
mid-flight, a different one each run.
Add cache=unsafe to each guest drive so guest flushes become no-ops
against the host page cache, removing the fsync-gated etcd stalls. This
is safe only because these CI VMs are ephemeral and destroyed every
run; never use it for real data.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit
|
||
|
Yan Bondarenko
|
6e5277937e |
fix(kafka): make topics[].config optional
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
6a87565438 |
fix(clickhouse): fetch tags via skopeo and fail closed on a missing pin
Address review of the version generator (hack/update-versions.sh):
- Fetch tags with `skopeo list-tags` (registry v2 /tags/list) instead of
the Docker Hub v2 API, whose anonymous pagination caps at 1000 entries
and then returns an error page with no `.results`, silently truncating
the list so the frozen 24.9.2.42 fell outside the window and the run
failed. skopeo is already in build-deps and is how the sibling
generators (postgres, redis, mongodb, opensearch, kubernetes) fetch.
- Make a frozen default tag missing from the registry a hard error
instead of silently falling back to a newer patch, which would move
the default image for existing installs.
- Restore 0644 on the regenerated files (mktemp's 0600 survived the mv).
- Exclude /hack from the chart tarball via .helmignore.
- Refresh files/versions.yaml: v25.8 -> 25.8.32.4, the true latest 25.8
patch now visible with the complete tag list.
- bats: add a missing-frozen-tag hard-error case and make the atomicity
test reach the resolution phase after a valid default rather than
failing at the default-in-set check.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Alexey Artamonov
|
768f29a235 |
feat(clickhouse): add configurable version parameter
Add a `version` value (v25.8, v25.3, v24.9; default v24.9) to the
ClickHouse chart, backed by files/versions.yaml and resolved by
templates/_versions.tpl for both the ClickHouse server and Keeper
images, so the Cozystack API returns the engine version (spec.version)
like the other managed-DB charts. Default v24.9 -> 24.9.2.42 keeps
existing installations byte-for-byte identical.
The maintainer-run hack/update-versions.sh generator intersects the
server/keeper Docker Hub tags byte-collated (LC_ALL=C) so comm keeps
tags across the X.9 -> X.10 boundary; freezes the default major to the
tag it already ships (read from versions.yaml, so the freeze follows the
default); errors on a dropped default or an unresolvable major; writes
atomically; and runs on stock macOS bash 3.2 / BSD. Covered by
hack/clickhouse-update-versions_test.bats (10 cases) and the
tests/version_test.yaml helm-unittest suite.
Refs: #1246
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit
|
||
|
Andrey Kolkov
|
22b1e074a5 |
fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
2cb8c743cb |
test(release): cover the split-map and repository-less ref shapes
No fixture in verify-promoted-packages_test.bats built a shape-3 split map -- `repository:` on one key, `tag: <tag>@sha256:<digest>` on another -- which hack/lib/image-refs.sh itself calls the dominant shape in the tree. The existing fixture's single-string ref carries a registry host, so its rc-to-stable rewrite normalizes through the other branch and passes whether or not host-less refs are handled, which is why the suite looked covered while a clean promotion could still be rejected. Add two cases. The first promotes a split map whose tag moves from rc to stable with the repository and digest held constant; reverting the normalization fix reddens it on "promotion changed the container repository/digest set", diffing v9.9.9-rc.3@sha256:eee... against v9.9.9@sha256:eee... -- the same tag-as-repository pair, with identical digests, that CI reported. The second pins the property a blunter fix would break. A digest-only `image:` map with no `repository:` (the kuberture shape) is seen by no rule but shape 1, so it must still contribute its digest: the case moves that digest and nothing else, and asserts the rejection comes from the digest set rather than a neighbouring leg. Replacing the fix with an outright `continue` over host-less refs makes the verifier exit 0 on that changed digest and reddens the case at its exit-status assertion. Negative assertions are counted rather than `!`-negated: under hack/cozytest.sh a `! grep -q` cannot fail, so it would be a comment shaped like an assertion. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
4fcc111142 |
fix(release): compare host-less image refs on digest alone
hack/verify-promoted-packages.sh reported "promotion changed the
container repository/digest set" for a v1.6.2 promotion that moved no
container bytes at all: all 48 real repo@digest refs and all 48 digests
were byte-identical between the rc artifact and the promotion candidate.
The six entries it flagged as changed repositories were not
repositories. They were bare tag strings -- v1.6.2-rc.1 on the rc side,
v1.6.2 on the candidate side -- pairing one-to-one by identical digest.
hack/lib/image-refs.sh emits two entries per shape-3 split map: the
correctly joined repository@digest, plus shape 1's recursive scrape of
the raw .tag scalar, which carries no repository. The library documents
that degeneracy and leaves host-less refs to its callers;
promote-retag.sh drops them, the verifier had only a
*/cozystack-packages exclusion. So `${image%:*}` -- a no-op on a string
holding no colon -- handed the tag over as the repository, and promotion
rewrites exactly those tags.
Set an empty repository for a ref with no `/` ahead of its digest, so it
compares on the digest, which is the whole container identity such a ref
carries and the only part promotion must not move. Dropping these refs
instead would be wrong rather than blunter: packages/system/kuberture
carries an `image:` map with a `tag:` and no `repository:`, so shape 1
is the only rule that sees its digest, and skipping it would silently
stop proving that digest unchanged.
Reproduced over the release-1.6 packages tree with the shape-3 tags
rewritten rc-to-stable: before, the six pairs above diverge; after, both
sides normalize to 60 identical entries, kuberture's digest included and
kube-ovn's real repository still visible through its shape-4 join.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
672a32b9df |
test(release): pin the promotion contract this branch has to satisfy
hack/verify-promoted-packages_test.bats is byte-identical to main: 13
tests driving the verifier against a mocked `flux`, so they need no
registry. They cover the fail-open shapes that matter most — an
unreadable file during the rc-reference scan, a collector that returns
nothing, a digest that moved while candidate and merge tree still match.
hack/promote-gate-contract.bats is NOT main's copy, and the difference is
deliberate. main's version pins a `parse` job, a `skip_e2e_gate` input,
promote-time changelog and website-docs jobs, and the `labeled`
pull_request trigger they depend on — none of which exist here, because
`Promote RC` is a workflow_dispatch and runs from the ref it is
dispatched from, which is main. Run verbatim against this branch it fails
on its first test and the runner aborts the suite.
So this pins the half release-1.6 actually owns — the producer side:
* an rc tag push calls e2e-tag.yaml with the pushed ref, only for
-rc. tags, and grants the static permission ceiling the called
workflow declares;
* the called job's name is the exact literal main's gate composes and
searches for, asserted with grep -F because `${{` as a basic regular
expression matches nothing and would make the pin pass by finding its
own subject absent;
* the suite it runs is the full one (empty CHAINSAW_SUITES), with the
Makefile target and the sandbox chainsaw binary it needs;
* all five files promote-rc.yaml's preflight reads off the target base
are present, plus the two libraries the verifier sources by relative
path;
* verify-release-candidate is keyed on author and head branch and
carries NO label condition, so restoring main's from muscle memory
switches the job off here and fails this test instead;
* the candidate verification precedes the stable tag, the submodule
tag, the release publish, the retag and the chart publish, and
follows the toolchain, the login and the tooling checkout it needs;
* the tag-time changelog is validated and ported, not regenerated.
Every pin was mutation-checked: breaking the thing it describes turns it
red.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
5673249d47 |
ci(release): validate the tag-time changelog and port it from the tag
Since the promote PR is based on release-1.6, the changelog it carries merges onto the maintenance line and never reaches main. The backstop job in tags.yaml only asks whether the file is on origin/main, so it sees "absent", spends an AI run, and produces text nobody reviewed — which update-releasenotes.yaml then pushes into the published release body, overwriting the reviewed notes the release actually shipped with. Ask the second question too. `at_tag` is whether the changelog is present in this checkout, which is the tag commit and therefore the promote merge commit. When it is, port that reviewed file to main verbatim and skip generation entirely; the AI now runs only when the changelog is genuinely absent on both sides. Add hack/validate-changelog.sh, byte-identical to main, and gate the commit on it. A ported file came from the tag and a generated one came from a step that is deliberately allowed to fail, so neither had been checked: the old guard only asked whether the file was non-empty, which a truncated Copilot stream satisfies. The validator asserts the H1, the release-link comment, the trailing compare link and at least one section, all carrying the expected version, so a fragment or a changelog generated against the wrong tag cannot reach a release body. Verified against this branch's own shipped changelogs: v1.6.1, v1.5.2 and v1.5.1 all pass, and v1.6.1 checked against 1.6.2 fails as it should. This takes effect for tags cut after it lands — tags.yaml and the script it calls both come from the tag's own tree. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e85482bacc |
ci(release): complete the candidate-aware promotion pipeline on release-1.6
Promote RC refuses to promote a release whose target base cannot verify
the packages candidate. It checks five markers on the base branch before
its first registry write and fails closed with "Target base
'release-1.6' lacks '<path>'". This branch satisfied one of the five
(hack/lib/image-refs.sh), so the next patch release off release-1.6
could not be dispatched at all.
Add the two halves the base has to carry:
* hack/verify-promoted-packages.sh and hack/lib/promoted-packages.sh,
byte-identical to main. The verifier pulls the candidate by its
committed digest and proves it matches the merge tree apart from the
artifact's impossible self-reference, that promotion rewrote every
version-line image tag, and that the container repo@digest set is
unchanged from the rc artifact.
* the two jobs that run it. verify-release-candidate in
pull-requests.yaml checks GitHub's prospective merge tree so a
packages/ edit pushed to the promote PR shows up as a failed check,
and a Verify stable packages candidate step in the finalize job of
pull-requests-release.yaml repeats it against the real merge commit.
Position is the whole point of the finalize half: it runs before the
write-once stable tag, the draft-release publish and the retag, so a bad
candidate is refused while nothing stable-named exists yet. The former
"Set up toolchain (skopeo, yq, helm)" and "Login to registry (GHCR)"
steps moved up with it and are no longer duplicated later in the job;
the toolchain now also installs flux and pins yq by version and sha256,
because an irreversible step must not depend on whatever
releases/latest/download resolves to today.
verify-release-candidate is keyed on the promote PR's author and its
head-branch shape rather than on the `release` label main matches. This
branch's on.pull_request.types has no `labeled` event, and the label is
applied about a second after the PR opens (a separate call in
gh pr create), so the `opened` payload carries no labels and a
label-keyed guard would never fire here.
.release-tooling is gitignored, matching main, so the trusted-base
checkout can never be staged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
e226b85f86 |
fix(ci): overlay images from the PR base branch, and publish per-line artifacts
Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.
The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.
Three deliberate choices:
* The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
per-release and rc staging branches promote-rc.yaml and tags.yaml create
(release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
images come from the tag build, and rebuilding them would be waste.
* WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
image and build-main.yaml is deliberately its only, serialized writer so
concurrent builds cannot race on the cache manifest (the 409 class #2711
fixed for image tags). A line build can overlap a main build, so writing here
would reintroduce that race. Line builds read the cache.
* A missing artifact still degrades to committed refs, but on a release line it
now emits a ::warning:: naming the branch. Silent degradation is
indistinguishable from a working overlay, which is how a mis-specified branch
filter would hide for a whole release cycle.
Cost: one `make build` per push to a maintained line, i.e. per merged backport.
hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
09f4cd10da |
fix(ci): only overlay current-main images on main-based PRs
The PR finalize job pulls cozystack-packages:main and repoints every package
the PR did not rebuild at current-main images. On a main-based PR that is the
whole point: it stops e2e from testing last-release images for everything
outside the build matrix. On a release-line PR it is wrong — the committed refs
there are not stale, they are that line`s released digests, and the charts are
written against exactly those.
So a release-line PR was installing main`s binaries onto its own charts, and the
mismatch grows with every commit main gains. #3437 is the demonstration: a
one-line change on release-1.6 that deactivates an app failed install
deterministically, twice, with
SchemaError(...core/v1alpha1.Option.spec): unknown model in reference:
"...core~1v1alpha1.OptionSpec"
from main`s cozystack-controller serving an aggregated OpenAPI that branch`s
charts cannot validate against. Nothing in the PR was broken; the lane was.
Left alone this makes every 1.6 backport look red, which is when release-branch
PRs are busiest.
Both overlay steps are now gated on `github.base_ref == main`, so a release-line
PR keeps its committed digests — the behaviour that predates the overlay.
Retargeting the overlay at a per-line artifact would be better but is not
possible today: build-main.yaml publishes only cozystack-packages:main, and the
registry carries no release-* equivalent (verified against the packages repo`s
tag list: `main` plus per-PR tags, nothing else).
hack/overlay-main-images_test.bats pins the wiring per step, so adding a third
overlay step without the guard fails the suite. Mutation-checked by removing one
guard. 12/12 green; actionlint clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
96a76cb2fc |
[Backport release-1.6] fix(release): make promote-retag digest verification media-type-agnostic (#3473)
# Description Backport of #3435 to `release-1.6`. |
||
|
Myasnikov Daniil
|
a4d73b2346 |
fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls
Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.
The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.
Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.
The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.
Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.
The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.
The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.
29/29 tests green; sh -n clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
4eb4948675 |
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors
hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (#3431).
Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.
Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.
Fixes #3431
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
ba87460aae |
fix(release): prove a stable tag is absent before copying over it
Review notes from #3435, and a correction to my first attempt at them.
manifest_digest gated on skopeo`s exit status alone, so a zero exit with no
bytes hashed the empty string to sha256:e3b0c442... — a digest that looks real
and belongs to no manifest. The first fix simply required non-empty bytes, which
was worse than it looked: the caller turns an empty result into "not published,
safe to copy", so a registry answering 200 with an empty body went from an
accidental refusal (the bogus digest mismatched, so the write-once check
refused) to an actual copy over whatever the tag really held. Verified against
real skopeo 1.23: a 200 with Content-Length 0 does exit 0 with no stdout.
The helper is now explicitly three-state. A digest, or empty output for a tag the
registry PROVED absent, or non-zero for anything indeterminate — which under
set -e aborts the promotion before it writes. Absence has to be reported as
"manifest unknown"; 429, 5xx, auth and network failures all exit non-zero with
no bytes too, and reading those as "unpublished" is the same fail-open one level
down. finalize retags ~42 refs with no retry wrapper, so a single rate-limit was
enough to reach that path. skopeo`s stderr is captured instead of discarded, so
the diagnostics now name the real failure.
Tests: the write-once refusal branch the review found untested (an existing
stable tag at a different digest must be refused before any write), the empty-
body case (refuses, writes nothing, and the empty-input hash never appears in
the diagnostics), and a 429 that must not read as an absent tag. The mock now
emits realistic registry errors, since a bare non-zero exit is no longer proof
of absence. Both new guards are mutation-checked: accepting any non-zero exit as
absence fails the 429 test, dropping the empty-bytes check fails the empty-body
test.
11/11 tests green, hack/image-pin-consistency.bats still 2/2, sh -n clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
Myasnikov Daniil
|
0686690381 |
fix(release): make promote-retag digest verification media-type-agnostic
skopeo inspect --format '{{.Digest}}' fails on OCI artifacts: for
cozystack-packages (config.mediaType
application/vnd.cncf.flux.config.v1+json) it prints nothing and exits
nonzero, so the v1.6.0 finalize aborted its post-copy verification right
after both copies had already succeeded, leaving 32 of 43 repos without
stable tags and skipping the cozy-installer publish. --dry-run never
executes this branch, so rehearsals could not catch it.
Compute the digest as sha256 of the raw manifest instead, which is how
registries define it and works for any media type. The helper gates the
hash on skopeo's exit status via a temp file rather than a pipeline:
a missing tag still yields an empty string (pre-check reads that as
"tag absent, proceed to copy"), never the empty-input hash
sha256:e3b0c442..., and never an abort under set -eu.
Verified against live GHCR: the raw-manifest hash of
cozystack-packages:v1.6.0 matches the digest finalize expected
(bf68208730860fa8...), and agrees byte-for-byte with the old method on
regular multi-arch images. New cozytest cases cover the OCI-artifact
digest, a post-copy mismatch, and the missing-tag path.
Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit
|
||
|
myasnikovdaniil
|
55195ded0c |
test(e2e): capture previous container logs and stop killing the snapshot (#3346)
## What this PR does Root-causing a recent E2E failure was nearly blocked because the diagnostics captured on failure had collected almost nothing. Two independent mechanisms were responsible, and both had been silently in place for a long time. **The failing container's previous logs were never captured.** The pod had crash-looped, so the decisive evidence lived in the first container instance, reachable only through `kubectl logs --previous`. Chainsaw's `podLogs` action has no `previous` field, so this could not be expressed per-suite; the conclusion had to be reconstructed from indirect evidence instead. A new capture runs from the global catch and from the BATS failure hook, selecting containers at runtime by `restartCount > 0` across both init and regular containers, so a healthy run stays quiet and future suites are covered without per-suite wiring. **The catch was structurally guaranteed to be killed.** A Chainsaw `script` op is bounded by `timeouts.exec`, which is 5m, while the catch's own inner budgets summed to roughly 960s. It reported `SCRIPT | ERROR | signal: killed` and produced nothing — and its second leg, the data-plane capture, had therefore never run at all. The kill signature was itself misleading: `timeout` exits 124 and cannot produce that message, which comes from Go's `exec.Cmd` killing the process group. Each catch op now carries an explicit timeout above the budgets it contains, so the inner one fires and leaves a partial, usable artifact instead of losing everything. **The snapshot was truncated at 60 seconds.** `crust-gather --duration` defaults to 60s and was never passed, so every snapshot ever taken was cut short, with nothing in the artifact saying so. The API discovery it performs first is not covered by that budget at all, which is where the outer wall-clock was going on a cluster with a degraded aggregated APIService — precisely the state a failed run is in. Both budgets are now set explicitly, and the collector's output goes to a log inside the snapshot with a completeness verdict, so "is this snapshot complete?" is answerable from the artifact. Failure paths are reported rather than swallowed: an unreachable API says so instead of implying nothing had restarted, a truncated capture is kept and marked, and every dropped item is logged, since silent truncation reads as full coverage. ### What is not verified here The mechanism claims are read from the pinned versions of Chainsaw and crust-gather. Everything cluster-facing is not verified by this PR: the go-template's output shape against a live API server, `--previous` returning the first instance for a real crash-looper, and whether the new collection budgets suffice on a degraded cluster. The bats suite covers the selection helpers only, and those cases are mutation-tested. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Diagnostics** * Improved end-to-end failure evidence by capturing “previous instance” logs for crash-looping pods (including restarted init containers) and prioritizing relevant namespaces. * Added time-bounded diagnostics collection with explicit outer timeouts; collector output is saved to per-snapshot log files. * Classifies collector outcomes as complete, timeout-truncated, or failed, while preserving the original test failure signal. * **Documentation** * Expanded E2E failure-diagnostics guidance with new timeout and evidence-capture conventions. * **Tests** * Added unit-style tests for previous-log selection, prioritization, row limiting, and deterministic log artifact naming. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d0c0f399f8 |
fix(ci): cover every image-ref storage shape in promote, retag and mirror (#3404)
## The bug Promotion rewrites the rc version substring to the stable version across the vendored image references. That rewrite lived inline in `promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus `packages/apps/kubernetes/images/*.tag` **only**, on the premise that the kubernetes app was the only one whose `.tag` files carry the cozystack version. Nine other `.tag` files do, and `system/multus` seds its reference straight into a vendored upstream daemonset manifest. #3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33 of 55 first-party references still reading `v1.6.0-rc.4`**. `hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind spot from the other direction — both scanned the depth-2 `values.yaml` alone, so those images were never retagged to the stable version (**30 references selected before, 42 after**) and never mirrored to the public registry for a nightly. A retag miss leaves an image short of a tag, since the digest still resolves inside one registry. A mirror miss is worse: the host rewrite walks the same file list, so a reference that is neither mirrored nor rewritten leaves the published tree pointing at the private build registry. `v1.6.0` is the first release to go through promotion — every earlier release was a full rebuild from the tag, which stamped every file uniformly. The glob has always been wrong; the old path just made it unobservable. ## Why it could not be caught before The rewrite was workflow-inline, so there was nothing to unit test. The only way to observe the miss was to cut a release. It is a script now — `hack/promote-rewrite-tags.sh` — and `hack/promote-rewrite-tags_test.bats` round-trips it against the real tree under `make unit-tests`, with no cluster, registry or release involved. ## The fix The three call sites each carried their own idea of where a reference can live, which is how they drifted apart. That knowledge moves into `hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()` for the files, `collect_image_refs()` for the references inside them — and all three source it, so a new storage location is declared once and reaches every consumer. The rewrite also asserts its own postcondition: it scans **wider** than it rewrites and fails the promotion when a reference survives in a location the enumeration does not know about, rather than shipping it quietly. The wide scan is shape-filtered to image-reference position, because `X.Y.Z-rc.N` is an ordinary version string that other things legitimately contain — `kubevirt-csi-driver/go.sum` pins `github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:` instead would have been wrong: `system/cilium/values.yaml` keeps `tag:` and `digest:` under separate keys. `docs/agents/image-refs.md` records the contract — three tag classes, three storage shapes, the invariants, and which consumer does what. ## Not a bug: kamaji `cluster-api-control-plane-provider-kamaji` stays at `v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild of an upstream component, versioned by that component rather than by the cozystack version line, so no rewrite is owed. A test pins this so a future fix here cannot over-reach. ## Documented known gaps, deliberately not fixed here Both are pre-existing and in the **host** dimension; the version dimension this PR owns is complete, because the version always lives in a `tag` key regardless of where the host sits. - **The nightly host rewrite cannot reach a split host.** It is a literal `<src-registry>/` substring replace, so `keycloak-operator` (host in a sibling `registry:` key) and `kubeovn` (`global.registry.address`, no trailing slash) are mirrored but keep pointing at the build registry in the published tree. Fixing it needs structure-aware rewriting. - **A reference inside a gzip is invisible.** `capi-providers-cpprovider` ships `files/components.gz`, and the promote postcondition cannot catch this class at all since `grep -rIl` skips binaries. Nothing is owed today — kamaji is component-versioned and the image is still mirrored and retagged via its `.tag` — so the residual is that a nightly's kamaji ConfigMap keeps the build-registry host. ## Testing - 17 tests in the new suite; `make bats-unit-tests` green (312 tests, exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow. - Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`, `1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no false negatives. - Every test is mutation-proven. Reverting the enumeration to the old glob, deleting any of the five YAML shapes, corrupting emitted digests, reverting extras to parse-only, dropping the unreadable-file guard, broadening shape 4 beyond `global.images`, or planting an unenumerated first-party reference in any storage shape each fail a named test. ## Reviewer note, out of scope `platform-migrations` is pinned at two different digests — `core/platform/values.yaml` at `v1.5.0` and `backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned pin; no Makefile writes that key, tracked in #3143). A `promote-retag.sh` dry-run emits **two copies to the same destination tag**, so the write-once guard will fail the promotion. This is independent of this PR and needs a decision about the backup-client image version, so it is left alone here — but it blocks the 1.6.0 promote and wants fixing before the re-cut. ## Release note ```release-note Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry. ``` Fixes the promote half of #3143. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Image promotion and nightly mirroring now detect digest-pinned image references across values files, standalone tag files, and supported templates. - Stable release preparation supports RC→stable tag rewriting via a dedicated promotion script, with broader verification to ensure no RC references remain. - **Bug Fixes** - Improved completeness of ref discovery and scanning, including previously missed template/tag locations. - **Documentation** - Added and linked guidance documenting image-reference invariants and how tooling enumerates supported storage shapes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
bbfe9f9a9d |
fix(backupstrategy-controller): give chBackupClientImage a producer (#3405)
## This is a workaround `backupstrategy-controller` should not be running the platform **migrations** image. This PR does not fix that — it makes the existing reuse *safe* so the v1.6.0 promote can proceed, and documents in the tree exactly what is wrong and what should replace it. Treat it as a stopgap with a TODO, not as an endorsement of the design. ## What is actually wrong The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its script uses exactly four binaries — `curl`, `jq`, `sleep`, `date`. For that it currently pulls `platform-migrations`: | | | | --- | --- | | Size | **349 MB** (a dedicated `alpine + curl + jq + ca-certificates` measures **23 MB**) | | Also contains | `kubectl`, `helm`, `git`, `cozyhr`, the `etcd-migrate` binary, every migration script, vendored etcd CRDs | | `ENTRYPOINT` | `run-migrations.sh` | Two consequences worth naming. A tenant-adjacent backup Pod carries cluster tooling it has no use for. And because the image's own entrypoint is the migration runner, the `command:` override in the strategy template is load-bearing — drop that one line and the Pod runs platform migrations with `kubectl` and `helm` already on the PATH. It also creates the coupling that caused the bug this PR fixes: bumping `ETCD_OPERATOR_VERSION` or editing any migration script re-digests the image, forcing the backup client's pin to move for reasons that have nothing to do with backups. ## The release blocker `platform-migrations` was pinned at two different digests in the committed tree: | Location | Pin | Producer | | --- | --- | --- | | `packages/core/platform/values.yaml` `.migrations.image` | `v1.5.0@sha256:8bf61f17…` | `packages/core/platform/Makefile` | | `backupstrategy-controller/values.yaml` `.chBackupClientImage` | `v1.4.0-rc.2@sha256:17390197…` | **none** | Promotion retags **by digest**: `hack/promote-retag.sh` copies every collected `<repo>@<digest>` to `<repo>:<stable-version>`. Two digests under one repository produce two copies competing for the same destination tag — the first wins, the second hits the write-once guard, and the promotion fails. A dry run against `main` emits exactly that pair: ``` ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:17390197… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ▸ ghcr.io/cozystack/cozystack/platform-migrations sha256:8bf61f17… → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9 ``` `v1.6.0` is the first release cut through promotion rather than a full rebuild. Every earlier release ran `make build`, which restamped both copies as a side effect and hid the missing producer. ## What this PR does `packages/core/platform/Makefile` now stamps both keys with the same ref, so they move in lockstep by construction rather than by instruction, and the committed value is aligned to the digest `core/platform` already carried. Moving the runner forward two minors is safe: it is used only as a curl + jq container, and the image built at `v1.5.0` carries them (`git show v1.5.0:…/migrations/Dockerfile` → `apk add … jq ca-certificates bash curl`). ## What should replace it A dedicated `ch-backup-client` image built by **this package's own Makefile** — measured at 23 MB, 15× smaller, with none of the cluster tooling and none of the entrypoint hazard. That also deletes the cross-package stamp this PR adds, because the pin would be produced by the package that owns it. The objection that has kept this reuse in place — "a second tag release CI would have to re-digest each cut" — does not survive contact with the actual defect. Every other first-party image already pays that cost, automatically. What broke here was not an extra tag; it was **a pin with no producer**, which a package-owned image cannot reproduce. I am deliberately not doing that here: it means a new image, a new build target and a new digest introduced during a release cut. It belongs in a follow-up. ## The guard `hack/image-pin-consistency.bats` asserts the general invariant — no repository pinned at more than one digest — driven through the real promotion selector so it tracks whatever set promotion actually acts on, rather than a reimplementation that could drift from it. Plus a specific lockstep assertion so a recurrence names its cause. This check could not have been added before now: it fails on the very duplicate this PR removes. ## Testing - `hack/image-pin-consistency.bats` — both tests pass; verified they fail when the drifted pin is restored. - `make bats-unit-tests` green (294 tests, exit 0). - `make -C packages/system/backupstrategy-controller test` — 11 tests, 3 suites, green. - `helm template … --set backupStorage.bucketNameOverride=test-bucket` renders the strategy Pod with the new ref. - The Makefile's `yq` step was simulated directly to confirm the relative path and key path resolve and that comments survive the in-place edit. - Merged locally with #3404 onto `main` and re-run: the expanded ref collector in that PR surfaces no additional duplicate pins (314 tests, exit 0). ## Relationship to other work Relates to #3143 — this is one instance of its unowned-pin class, the one its discussion flagged as having no producer. The four never-built packages there remain untouched. Independent of #3404 (promote/retag/mirror storage-shape coverage); the two touch no common files and can merge in either order. Both are needed before the `v1.6.0` promote is re-cut. ## Release note ```release-note Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Ensured the platform migrations image digest is pinned consistently between the platform and backup strategy components. * Updated the backup strategy component to reference the new pinned `platform-migrations` digest. * **Tests** * Added promotion/retag consistency checks to fail on conflicting source digests for the same stable tag. * Added drift detection to ensure `platform-migrations` image references match across consumer configuration files. * **Documentation** * Expanded in-chart comments and values documentation clarifying the coupling and workaround behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey KolkovandClaude Opus 4.8
|
4605bbf3db |
test(mariadb): pre-clean the backup roundtrip against stale tenant-root state
Address review feedback on #3384. The TIA rule for
examples/backups/<app>/ that an earlier revision of this commit added
has since landed on main independently (
|