On the kube-ovn variants Cilium allocates no pod addresses, but the agent
still takes its router IP and the Gateway API Ingress IP from its
allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which
the documented installs slice from the same range kube-ovn allocates pods
from, so kube-ovn sooner or later hands a pod an address Cilium already
holds.
Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a
/29 per node from the last /21 of networking.podCIDR, and kube-ovn
excludes that /21 through --default-exclude-ips. The block stays inside
the pod CIDR on purpose. Envoy reaches Gateway API backends from the
node's Ingress IP, and kube-ovn masquerades host-originated traffic only
when the source lies in one of its subnets, so a block outside them such
as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes.
A post-upgrade hook moves existing clusters. It first checks that the
block has a /29 for every node and lies inside kube-ovn's default
subnet, and stops without changing anything otherwise. It adds the block
and every node's current Ingress IP to excludeIps of that subnet,
because the agent keeps a former Ingress IP in its ipcache as
reserved:ingress. It then waits until kube-ovn stops offering the block
and stops if an address in it is still allocated. kube-ovn leaves
excluded addresses out of the subnet status, so that check reads its IP
objects. Then, one node at a time, it clears the inherited /24 from the
CiliumNode, waits for the operator to allocate from the block and
restarts the agent.
networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn
variants, and the render fails otherwise. A /21 cut into /29s also caps
these variants at 256 nodes.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit dc362fa591)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Backport of #4828 to `release-1.6`.
PR lane on this line runs on Talos containers since the backport of
#4020, but `e2e-tag.yaml` (the rc gate that `tags.yaml` calls for every
rc from `release-1.6`) and `nightly.yaml` still boot QEMU guests. Every
rc tag run here since v1.6.2-rc.1 is red. v1.6.4-rc.1 failed with
kubernetes-latest and kubernetes-previous spending about 32 minutes each
at node-join deadline
(https://github.com/cozystack/cozystack/actions/runs/36124680874), while
the same 41 suites passed on container lane in the v1.6.4 promotion PR,
in a 93 minute job
(https://github.com/cozystack/cozystack/actions/runs/36383277279).
e2e plumbing on `release-1.6` diverged from `main` after #4020, so the
cherry-pick of the first commit has conflicts resolved against this
branch, and there is one prerequisite commit:
- `e2e-tag.yaml` and `nightly.yaml` get the same host module step,
`prepare-env-container`, endpoint and storage class env and two-part
teardown as `pull-requests.yaml` here. `e2e-tag.yaml` still requires the
release to carry its nocloud disk but doesn't download it anymore, and a
dispatch against a tag cut before container lane (v1.6.3 and older)
fails early and says to dispatch with `--ref <tag>`.
- nightly moves to 32-vCPU runner, because `hack/e2e-container-up.sh`
refuses a 24-vCPU host (`main` did the same earlier in db2ddfab2).
Tests: `platform-packages_test.bats` checks container lane env on all
three lanes, `promote-gate-contract.bats` covers the new guard (on
`main` it lives in `collect-images_test.bats`, there is no such file
here).
Differences from `main`: every lane keeps `timeout-minutes: 180`, the
follow-up comment fix about 215-minute budget is not included (this
branch has no such comment and no such value), fork lane wording is
dropped because there is no fork split here, docs edits cover only
paragraphs that exist on this branch.
Coverage lost on this line: DRBD and the `replicated` StorageClass,
Cozystack Talos node image with its extensions, CDI stock 600M worker
memory limit. Docs and comments that pointed at the rc run as that
coverage now say so. `nightly.yaml` runs only from `main`, so the change
there is for parity.
`tags.yaml` takes `e2e-tag.yaml` from the commit of the rc tag, so this
has to merge before v1.6.5-rc.1 is cut. If rc is cut first, dispatching
`e2e-tag.yaml` from `release-1.6` with that tag gives the same evidence.
`make bats-unit-tests` passes locally (487 ok, base has 486), but the
workflows themselves were not run yet. This stays a draft until
`e2e-tag.yaml` dispatched from this branch against v1.6.4-rc.1 is green,
I will put the run link here.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
ci(e2e): on release-1.6, the release-tag e2e gate and nightly run on Talos containers instead of QEMU guests. DRBD, the replicated StorageClass and the Cozystack Talos node image are no longer covered by any e2e lane on this line.
```
## What this PR does
Backport of the LINSTOR severity fix and its alert-rule guard from #3799
to `release-1.6`, together with the guard's follow-up #4812. Seven
piraeus-datastore rules use `warn`, which the default Alerta alarm model
does not accept, and they now use `warning`. Alert names, expressions
and thresholds stay unchanged.
Cherry-picked `3ade16a380c9d03b83ff28ea02af3b800fdd1b0f` and
`bd2cc0a749531433a237a9c8a6e21a6f21dee200` with their authors and
cherry-pick references. The KEDA severity commit from the same main PR
is left out because the KEDA rule it fixes does not exist on
release-1.6. The follow-up from #4812 is cherry-picked too. It makes the
guard catch a quoted kind or a kind followed by a comment, and fail on a
rule group it cannot see into once template actions are stripped.
### Validation
- All nine checks in `hack/alert-severity-contract.bats` pass with
`hack/cozytest.sh` on this branch. The wider discovery finds the same 39
rule files as before and no rule group comes out empty.
- Against `release-1.6` at `da98ddc08` without the fix, the same
contract fails on exactly the seven `warn` rules corrected here.
- Delivery through Alertmanager and Alerta was not exercised.
### Screenshots
Not a UI change.
### Downstream repositories
The file-level trigger map was checked against both changed files: a
rule file under `packages/system/piraeus-operator/alerts/` and a new
test under `hack/`. No existing tool, make target, API schema, package,
output resource or documented workflow is renamed or changed. No
companion repository update is required.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(linstor): backport accepted warning severity for seven piraeus-datastore alerts to release-1.6, with a regression guard for first-party alert rules
```
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.
Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.
DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit da28ea7ee8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The guard found rule files by an exact `kind: PrometheusRule` or
`kind: VMRule` line, so a quoted kind or one followed by a comment took the
file out of the check. A rule group whose rules all come from template
actions is empty once those are stripped, so it passed with nothing
checked. Discovery now accepts those spellings, and a rule document without
groups, or a group without a visible alert or recording rule, fails the run
the way a file that cannot be parsed does.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit dc814f3ae5)
Alerta refuses an alert whose severity is not in its alarm model, so a
rule with such a label is dropped every time it fires and the alert never
reaches anyone. Seven piraeus-datastore rules used warn and the keda
at-max rule used info, and nothing in the tree caught either.
The contract collects every PrometheusRule and VMRule document under
packages, in alerts files and in chart templates, and fails on any
alerting rule whose labels.severity is missing or not in the accepted
list. Helm actions are stripped before parsing, so a templated rule is
still checked, and a file that cannot be parsed fails the run instead of
being skipped. Values are compared as whole words.
none stays out of the list because the Alertmanager config routes it to
a blackhole receiver. Vendored charts under packages/**/charts/** are out
of scope because they are re-vendored rather than edited.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
(cherry picked from commit bd2cc0a749)
Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
etcd.aenix.io cluster with etcd-migrate and then scales the operator
back to 1. At that point it is still the legacy operator, and it has
nothing left to manage. Until a tenant's etcd HelmRelease reconciles the
new chart, Flux can upgrade it with the old one, which re-creates the
legacy EtcdCluster that etcd-migrate just orphan-deleted. The legacy
operator then reconciles it, re-creates the legacy StatefulSet and
rewrites the legacy etcd-headless Service, dropping the owner references
etcd-migrate pointed at the adopted EtcdMembers. When the new chart
prunes the legacy EtcdCluster, the garbage collector deletes the Service
with it, and the adopted members' etcd-<i>.etcd-headless names stop
resolving.
After a successful adoption the operator now stays at 0 until the
etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade. A failed or interrupted adoption
still restores it through the EXIT trap.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 6248d458ad)
# Description
Backport of #4253 to `release-1.6`, together with #3460 which it depends
on. On 1.6 kube-ovn chart and image still come from
cozystack/kubeovn-chart, so draft Dockerfile had nothing to build it and
kubeovn build failed (no `image` target). #3460 moves build in-tree on
same kube-ovn 1.15.10 this branch runs, and brings nightly-mirror fix
for kube-ovn split registry host that becomes needed once kube-ovn is
built by CI. Bot draft also dropped second commit of #4253
(`fix-vmim-target-pod-retry.diff`), it is back now.
I built image locally, all three patches apply to kube-ovn v1.15.10 and
`TestVMIMigrationSchedulingRetry` passes.
The CAPK build suite came over from main asserting that an image build
under OCI_EXPORT_DIR writes an OCI archive with --push=0. That export
path lives in hack/common-envs.mk on main and was never brought to this
branch, where every build pushes, so the suite failed here on every
run.
What the suite is for still holds on this branch: the build stamps a
digest-pinned ref into the .tag file that the image-ref tooling finds,
it leaves the shipped components untouched, and a failed build or a
missing digest keeps the previous pin. Those checks stay; the OCI
parameter and the three archive checks go.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/bats-no-exit-trap.bats is an exact ratchet, and the new suite installs
three traps inside explicit subshells without declaring them, so
make unit-tests fails on the merge commit. The header now declares the
count and says why the three are not debt.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 0e4d6a305c)
The nightly host rewrite is a literal "<src-registry>/" substring
replace, so it only reaches a host sitting contiguously in front of the
repository. kubeovn keeps its host in global.registry.address with the
bare repository in a sibling key: no trailing slash, no match. The
image was mirrored to the public registry and the tree kept pointing at
the private build registry.
The gap was dormant while kubeovn's image came from a separate
repository, because the baked ref carried the public host and the
ownership filter skipped it. Building the image in-tree stamps the
build-registry host into that key and turns it live, so fix it here.
Add an end-of-line-anchored expression that fires only when the source
registry is the complete value; a contiguous ref continues past the
host, so the two expressions cannot both fire on one reference.
keycloak-operator splits at a different boundary and stays unfixed --
neither of its keys holds the source registry as a whole. Documented,
and dormant while no CI path rebuilds it.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 6cb579338b)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Both suites matched "strimzi.io/cluster=kafka-test" as a bare substring,
and that is a prefix of the sibling release kafka-test2 and of
kafka-test-topics. Rendering the hook with either selector left both
suites green, so neither pinned the scoping they exist to pin. Anchor on
the leading "-l " and on the trailing delimiter; both variants now red in
both suites, and the six earlier hook mutations still red exactly where
they did.
Drop the notMatchRegex on a leading slash. It cannot fail on its own: the
anchored image pattern above rejects a leading slash first, and an empty
_cluster.images-registry renders no slash to begin with. Its rationale
moves into that pattern's comment.
Record why activeDeadlineSeconds and the 180s wait have to move together.
A deadline that fires mid-wait kills the Job, and with backoffLimit: 0 the
hook then returns non-zero, which is the wedged uninstall the deliberate
exit 0 on the timeout branch exists to avoid.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 637098f222)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The suite used bats' run helper. make bats-unit-tests does not invoke the bats
binary: it runs hack/cozytest.sh over every hack/*.bats, and that runner
implements @test and little else, so the file died at 'run: command not found'
and took make unit-tests with it. Under bats it was 5/5, which is where the 5/5
in the previous commit came from. Save $? by hand instead, the shape
app-cleanup-hooks_test.bats already uses; verified under both runners.
The fake kubectl no longer defaults DELETE_RC to 0. An unset value made every
failure-path assertion vacuous; now it exits 64 and says so. Cover both
RESTMapper wordings for an absent CRD rather than just the first.
Widen the image pattern to v[0-9]+\.[0-9]+(\.[0-9]+)? like harbor and mariadb:
pinning v1\.32 reintroduced the breakage the digest was made pattern-based to
avoid, one Renovate bump later. Assert the _cluster.images-registry override
and the no-leading-slash default, neither of which was covered: the hook is the
one image in this chart whose pull failure surfaces at uninstall.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 415aa82409)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The hook's exit code is the whole contract: a non-zero one returns from
Uninstall.Run before any release resource goes, keeping the topic operator
alive; a zero one lets Helm remove it. delete_test.yaml matches manifest text,
so it stayed green when the timeout branch was put back to exit 1, when the
CRD-absent branch was, and when the selector was widened past this release.
Add a bats suite in the shape hack/app-cleanup-hooks_test.bats already uses for
qdrant and harbor: render the script, run it against a fake kubectl, pin the
exit code of each branch. Each of the three mutations above now turns it red.
Drop hook-succeeded from the delete policy and reclaim the Job with
ttlSecondsAfterFinished instead. The script exits 0 on the degraded path too,
where topics still hold the finalizer, so deleting on success removed the only
record of that seconds after it was written. Assert both in delete_test.yaml.
Pin the hook image by shape rather than by literal digest: Renovate rewrites
digests under packages/*/templates/hooks/ and nothing rewrites tests/, so the
literal turns the next bump red for no defect.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 3326e0fb9e)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The post-delete absence assertions could pass vacuously if the two
release topics were never created. Asserting both topics first makes
the E2E prove that the pre-delete cleanup actually removed them.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 9d2fddd4e6)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vminstance suite waits for each VMDisk's DataVolume to be Ready
before anything consumes it. On `replicated` that works because the
class binds Immediate; on the container lane's `local` class, which is
WaitForFirstConsumer, CDI starts no importer for a disk nothing has
consumed yet, so the DataVolume sits in PendingPopulation and the suite
times out with no error anywhere. main makes the vm-disk chart request
immediate binding for every source; the chart on this branch requests it
only for uploads, and the lane must not change what it ships.
So on the container lane, right after each VMDisk is applied, the suite
waits up to 180s for CDI to create the DataVolume and its claim and
annotates both with cdi.kubevirt.io/storage.bind.immediate.requested,
the same request the chart already writes for uploads. Both, because
the DataVolume decides what phase CDI reports and the claim is what the
populator reads to start the import without a consumer. The QEMU lane
annotates nothing.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
CDI's own 600M worker memory limit was seen to OOM the decompress and
convert of a tenant worker disk near 100% on the container lane, after
which CDI retries from scratch and the DataVolume cycles forever rather
than failing. The same import completes at 600M on QEMU nodes, so what
needs the headroom is the substrate. main passes the raise through a
kubevirt-cdi chart value that this branch does not have.
The last install test now merges the same figures into the CDI CR on
the container lane: 4Gi and 256Mi for memory, CPU left at CDI's 750m and
100m because the failure was memory. It has to be the CR, not CDIConfig,
because the operator reconciles CDIConfig.spec from the CR and reverts a
direct edit, and the test waits for CDIConfig's published default to
read 4Gi, since that is what CDI hands its worker pods. The QEMU lane
keeps CDI's defaults.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
On the container lane every volume is on `local`, which is node-pinned
and WaitForFirstConsumer. Without --strict-topology external-provisioner
passes every topology segment as requisite and only prefers the node the
scheduler picked, so linstor-csi may provision elsewhere when that node
is full. A CDI importer mounts two such volumes and the scratch one is
created only after the pod is scheduled, so it can land on another node
and leave the importer unschedulable forever with nothing erroring.
main turns the flag on in the linstor chart; the chart on this branch
cannot, and the lane must not change what it ships.
So the last install test, on the container lane only, adds the same
csi-provisioner override to the live LinstorCluster, waits for
piraeus-operator to carry it into linstor-csi-controller and for that
rollout to finish, and asserts the flag on the Deployment. `args` has no
patchMergeKey, so the list is restated in full as piraeus-operator
v2.10.2 renders it. It runs last because the earlier tests that change
platform values upgrade linstor, and an upgrade re-renders the
LinstorCluster without the override. For the same reason the tenant
Kubernetes suites and both vminstance Tests re-check the flag before
they import, so an override lost to a later upgrade fails by name rather
than as an import stuck at ImportScheduled.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
The linstor chart adds a drbd-logger sidecar to every satellite, and on
the container lane's shared kernel there is no DRBD for it to tail, so
it exits at once. The satellite pod is then never Ready, piraeus never
registers the node, LINSTOR reports no nodes and no PVC binds. main
drops the sidecar through a chart value; the chart on this branch has
none, and the lane must not change what the chart ships.
So post-install prep applies a LinstorSatelliteConfiguration of its own
on the container lane, ahead of the linstor waits, with the same
DaemonSet patch main renders: delete drbd-logger from the generated
satellite DaemonSet. The two DRBD-only initContainers are already gone
through the chart's Talos satellite configuration. Once all three nodes
are Online, prep reads the live DaemonSets back and fails if a
satellite still carries the sidecar or none was found at all, so a
configuration that silently stopped applying cannot pass for one that
worked. The QEMU lane applies nothing and keeps the chart as shipped.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
The three checks that System-mode artifacts are gone after the mode
switch were `if kubectl get ... >/dev/null 2>&1`, which reads every
non-zero exit as "the object is gone". An RBAC denial, a timeout or a
missing CRD produces the same exit, and stderr went to /dev/null, so the
suite reported a clean System teardown for never having observed one.
--ignore-not-found is what separates absent from unaskable: absent is
exit 0 with empty output, everything else stays non-zero. Preferred over
matching NotFound in the message, which depends on wording and, under a
runner with tracing on, captures the trace instead of the error.
Each probe is pinned individually rather than as a group: the new case
breaks one query per iteration while the other two answer absent, so
reverting any single probe to the bare form reddens the suite. Verified
by doing exactly that three times. A test that broke all three at once
would have passed on any one surviving probe, which is how the first
attempt at this test fooled me.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit b30c7efaf6)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
.chainsaw.yaml states the rule and says why: an op that expires before
its inner bounds do is a SIGKILL on the process group, which takes out
the diagnostics the failure branch exists to print, while an inner
timeout leaves a partial capture and lets the remaining steps run.
refresh-and-assert-full-health, added on this branch, breaks it. The
poll deadline is 270s, the last in-loop read can start just under it and
run 25s, and the failure branch then reads again for another 25s: 320s
of inner bounds under a 5m op. Raised to 6m.
verify-processes-and-security-context breaks it harder and did so
before this branch: 240s for the process counts, then 120s for
connectionString and 120s for generations.reconciled, sequentially, is
480s under the same 5m. Fixed here rather than filed, because it is the
same defect one step away, found by applying the rule to the neighbour.
Raised to 9m.
The kubernetes-latest ceiling gets the measurement instead of a change.
The OIDC lifecycle folded onto that cluster adds up to 1m + 600s + 1m of
passing-path waits that the 67m op does not price, and raising the op is
not available: a guard holds COZY_OP_CEILING equal to both suites'
timeout, and two suites at 79m plus their teardowns exceed the job's
215-minute cap. On the green run of this branch the whole Test took
569.72s against the 4020s op, with kubernetes-previous at 470.19s. The
ceilings only bind once something is already hung, which is what the
existing residual note (#3666) says; now it says it with a number.
Adapted for release-1.6: the kubernetes-latest op here is 40m and no
guard ties it to another figure, so its note states what it does not
price instead of citing the COZY_OP_CEILING guard, the 67m ceiling and
the main-branch measurement, none of which exist on this branch.
Assisted-By: LLM
(cherry picked from commit b0e0f38fbe)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The readiness gate resets its stability window when a list read fails,
and nothing held that: the existing "transient list error" case fails on
the first read, when there is no window to invalidate, so removing the
reset left the suite green. The new case breaks a read mid-window, where
a gate that kept counting through the gap would accept a release set it
never observed stable across it.
The parallel fan-out contract asked make for --output-sync, which arrived
in GNU Make 4.0 while macOS still ships 3.81 -- so on a contributor's
machine the suite died on an unknown option instead of reporting the
contract. Guarded by version, with the reason in place. CI runs 4.x, so
the contract is still enforced where it gates a merge.
Adapted for release-1.6: the --output-sync guard is dropped together
with hack/unit-test-parallelism.bats, which this branch does not have
because the unit-test fan-out it covers is not backported.
Assisted-By: LLM
(cherry picked from commit 8456facc79)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two ways the gate did not do what its arguments promise.
The list read carried no bound. kubectl defaults to no request timeout,
and the read sits above the deadline test at the bottom of the loop, so
against a wedged apiserver it blocks there and timeout_seconds stops
being a ceiling: what arrives is the job timeout as a kill, instead of
the not_ready dump this function exists to print. Bounded with kubectl's
own --request-timeout rather than a `timeout` wrapper, because the unit
suite substitutes a kubectl shell function and `timeout` execs a binary.
The final diagnostic read is bounded the same way.
`now` was sampled before the read, so a read costing more than the poll
interval was priced at zero against the deadline and bought the loop
another whole iteration past it. Sampled after, on both branches.
And the count >= minimum_count half of the gate was pinned by nothing:
every fixture returned 11 or 12 items against a minimum of 11, so the
condition could be deleted outright and all five tests stayed green.
That threshold is the direct replacement for the `wc -l` existence
backstop this branch removes from hack/e2e-install-cozystack.bats -- the
only thing separating "everything is Ready" from "almost nothing got
created", since three HelmReleases that exist are all Ready. Both new
cases were checked by reverting the code they cover.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit ec7f17299b)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Adapted for release-1.6: this branch has no e2e-fork.yaml, no soft-red
gate, no ghcr.io mirror and no backup e2e, so those hunks are dropped
together with the tests for helpers that exist only on main; e2e-tag.yaml
stays on QEMU. The platform Package generator carries the API endpoint
and nothing else, because the linstor drbd.enabled and kubevirt-cdi
importerResources overrides need chart values this branch does not ship;
the container lane gets e2e-side substitutes for them in later commits.
run-kubernetes.sh takes the storage-class helper, the local-lane pool
baseline, the scoped drain and the OIDC fold, while cleanup keeps its
best-effort `|| true`. The OIDC suite references in select-install.sh and
the chainsaw README go with the suites here rather than in a later commit.
(cherry picked from commit 186547d835)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
verify_storageclass_fallback_default deletes the `replicated` class,
renders with --dry-run=server, and restores it inline "before any
assertion exit" -- the comment's words. Under the caller's `set -eu` the
render is a plain assignment, so a failed render exits the function
there: before rc is read, and before the restore runs. The management
cluster is then left with no default StorageClass for every suite after
it, which is the opposite of what the comment promises.
Captured with `|| rc=$?`. Pre-existing shape, but the restore block is
edited on this branch anyway, and the QEMU lanes are where it bites.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit b131bd56a4)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Backport of #3579 to `release-1.6`.
Tenant backend Deployment in the kubernetes suites still has one 300s
budget for scheduling and image pull on this branch. Right now those
suites on 1.6 mostly fail earlier, on node-join, so this starts to
matter once #3455 widens that window.
Cherry-picked clean with `-x`.
### Testing
- `make unit-tests` green, POSIX sh sweep clean.
```release-note
NONE
```
Backport of #3715 to `release-1.6`.
Both linstor waits in `hack/e2e-post-install-prep.sh` still run off one
15m deadline on this branch, so a slow linstor HelmRelease eats the
Deployment's budget and install reports the wrong link as broken. Each
wait now gets its own budget.
Cherry-picked clean with `-x`.
### Testing
- `make unit-tests` green, POSIX sh sweep clean.
```release-note
NONE
```
Backport of #3750 to `release-1.6`.
Sandbox Talos config and platform install on this branch still both use
`10.244.0.0/16`, so kube-ovn and cilium allocate from one range and
cilium refuses pod sandboxes with `IP ... is already in use`. It hit
`mariadb-2-backup-roundtrip` on the E2E run of #4328 here.
Cherry-picked clean with `-x`.
### Testing
- `make unit-tests` green, POSIX sh sweep clean.
- Real check is E2E on this PR.
```release-note
NONE
```
The tenant backend Deployment had a single 300s budget to reach
condition=Available, and two unrelated variable costs shared it. The
first is scheduling. What the suite establishes before that point is two
tenant nodes Ready, which is weaker than schedulable: a Ready node still
carries node.cilium.io/agent-not-ready until the tenant cilium agent
claims it, and a node the bringup has not finished with is
SchedulingDisabled. Scheduling took 2m18s and 1m57s in the two tenant
suites of one run, leaving the image pull to finish inside what was
left. It did not, and both suites failed with the same message for two
different shortfalls.
Wait for a node that actually accepts a toleration-free Pod before
creating the workload, on its own budget and its own failure message.
The gate encodes the scheduler's rule for such a Pod (Ready, not
unschedulable, no NoSchedule or NoExecute taint) over a custom-columns
probe, prints the node table on both outcomes so a timeout names the
taint that held it, and treats a failed probe as not-schedulable so an
API blip cannot release it. The readiness wait keeps its 300s, now
starting from a schedulable node, and dumps deployment, pod and event
state when it runs out.
Pin the workload image by digest. The tenant workers reach no registry
mirror, so nginx is pulled from Docker Hub on every run either way; the
digest fixes what that pull returns instead of leaving a floating tag
free to change size and content under a fixed deadline.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 5542ae52b7)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.
Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.
Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.
It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.
A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.
The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.
Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.
The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.
The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 86674c7084)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The sandbox Talos config and the platform install step both used
10.244.0.0/16, one as the Kubernetes cluster CIDR and the other as the
kube-ovn POD_CIDR, so two allocators drew from one range.
kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium is
chained behind it and allocates no pod addresses, but it still carves
its own per-node router IP and Ingress IP out of Node.spec.podCIDR,
which kube-controller-manager slices from the cluster CIDR. Sooner or
later kube-ovn hands a pod an address cilium already holds. When that
address is the Ingress IP the agent refuses the sandbox with "IP
ipv4:<addr> is already in use", and cilium never relents on it, so the
pod only proceeds if something reschedules it onto a different address.
In the failure this comes from, the same address was refused seven
times across six pod UIDs before the suite ran out of budget. When it
is the router IP nothing complains at all and only host-to-pod traffic
breaks.
Move the cluster CIDR to 100.65.0.0/16, which no other range in the
tree uses. That deliberately stops the sandbox matching the layout the
install documentation prescribes, so it is not a value to put back.
Pod addresses are unaffected: they still come from kube-ovn out of
10.244.0.0/16, and cilium does not require a chained endpoint's address
to sit inside its own allocation CIDR.
Excluding the addresses on the kube-ovn side would not work, because
cilium picks its offsets by random scan over the node's range.
The new guard checks the invariant rather than the literals, so either
value may move as long as they stay disjoint. It covers the service
and join ranges too, since landing on one of those instead is the next
way to get this wrong.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
(cherry picked from commit 9ace63b441)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The kubernetes-latest and kubernetes-previous suites flaked on the
worker node-join wait under host load. In run 30260770694 both failed at
exactly 12m with zero tenant Nodes registered and the tenant cilium HR
still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed
the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m
into this wait and the guests then had ~10m to import Talos, boot,
register a kubelet and let cilium turn the nodes Ready.
This single deadline absorbs the entire worker bring-up (the
machinedeployment replicas=2 gate clears before the VMIs even exist), so
under storage pressure 12m is too tight. A less-loaded fleet run passed
both suites unchanged, confirming load-induced slowness rather than a
stuck bring-up. Widen the deadline to 18m, which restores margin and
still sits well inside the 40m chainsaw step timeout.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 330acfe6e2)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The e2e QEMU guests boot with three raw virtio -drive lines that carry
no cache= option, so QEMU defaults to cache=writeback and honors every
guest fsync. The management-cluster Talos etcd fsyncs on each Raft
commit, while containerd unpacks dozens of images and CDI imports
multi-GB VM disks onto the same host-backed raw files during the
VM-heavy suites. The fsync storm serializes etcd's writes into
multi-second stalls, the mgmt apiserver times out lease PUTs, and many
operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji)
lose leadership at once — failing whichever chainsaw suite is
mid-flight, a different one each run.
Add cache=unsafe to each guest drive so guest flushes become no-ops
against the host page cache, removing the fsync-gated etcd stalls. This
is safe only because these CI VMs are ephemeral and destroyed every
run; never use it for real data.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit ad35afa534)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review of the version generator (hack/update-versions.sh):
- Fetch tags with `skopeo list-tags` (registry v2 /tags/list) instead of
the Docker Hub v2 API, whose anonymous pagination caps at 1000 entries
and then returns an error page with no `.results`, silently truncating
the list so the frozen 24.9.2.42 fell outside the window and the run
failed. skopeo is already in build-deps and is how the sibling
generators (postgres, redis, mongodb, opensearch, kubernetes) fetch.
- Make a frozen default tag missing from the registry a hard error
instead of silently falling back to a newer patch, which would move
the default image for existing installs.
- Restore 0644 on the regenerated files (mktemp's 0600 survived the mv).
- Exclude /hack from the chart tarball via .helmignore.
- Refresh files/versions.yaml: v25.8 -> 25.8.32.4, the true latest 25.8
patch now visible with the complete tag list.
- bats: add a missing-frozen-tag hard-error case and make the atomicity
test reach the resolution phase after a valid default rather than
failing at the default-in-set check.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit e707475f96)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
Add a `version` value (v25.8, v25.3, v24.9; default v24.9) to the
ClickHouse chart, backed by files/versions.yaml and resolved by
templates/_versions.tpl for both the ClickHouse server and Keeper
images, so the Cozystack API returns the engine version (spec.version)
like the other managed-DB charts. Default v24.9 -> 24.9.2.42 keeps
existing installations byte-for-byte identical.
The maintainer-run hack/update-versions.sh generator intersects the
server/keeper Docker Hub tags byte-collated (LC_ALL=C) so comm keeps
tags across the X.9 -> X.10 boundary; freezes the default major to the
tag it already ships (read from versions.yaml, so the freeze follows the
default); errors on a dropped default or an unresolvable major; writes
atomically; and runs on stock macOS bash 3.2 / BSD. Covered by
hack/clickhouse-update-versions_test.bats (10 cases) and the
tests/version_test.yaml helm-unittest suite.
Refs: #1246
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 3c78c0be56)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5bcdb8cfb8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
No fixture in verify-promoted-packages_test.bats built a shape-3 split
map -- `repository:` on one key, `tag: <tag>@sha256:<digest>` on another
-- which hack/lib/image-refs.sh itself calls the dominant shape in the
tree. The existing fixture's single-string ref carries a registry host,
so its rc-to-stable rewrite normalizes through the other branch and
passes whether or not host-less refs are handled, which is why the suite
looked covered while a clean promotion could still be rejected.
Add two cases. The first promotes a split map whose tag moves from rc to
stable with the repository and digest held constant; reverting the
normalization fix reddens it on
"promotion changed the container repository/digest set", diffing
v9.9.9-rc.3@sha256:eee... against v9.9.9@sha256:eee... -- the same
tag-as-repository pair, with identical digests, that CI reported.
The second pins the property a blunter fix would break. A digest-only
`image:` map with no `repository:` (the kuberture shape) is seen by no
rule but shape 1, so it must still contribute its digest: the case moves
that digest and nothing else, and asserts the rejection comes from the
digest set rather than a neighbouring leg. Replacing the fix with an
outright `continue` over host-less refs makes the verifier exit 0 on that
changed digest and reddens the case at its exit-status assertion.
Negative assertions are counted rather than `!`-negated: under
hack/cozytest.sh a `! grep -q` cannot fail, so it would be a comment
shaped like an assertion.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages.sh reported "promotion changed the
container repository/digest set" for a v1.6.2 promotion that moved no
container bytes at all: all 48 real repo@digest refs and all 48 digests
were byte-identical between the rc artifact and the promotion candidate.
The six entries it flagged as changed repositories were not
repositories. They were bare tag strings -- v1.6.2-rc.1 on the rc side,
v1.6.2 on the candidate side -- pairing one-to-one by identical digest.
hack/lib/image-refs.sh emits two entries per shape-3 split map: the
correctly joined repository@digest, plus shape 1's recursive scrape of
the raw .tag scalar, which carries no repository. The library documents
that degeneracy and leaves host-less refs to its callers;
promote-retag.sh drops them, the verifier had only a
*/cozystack-packages exclusion. So `${image%:*}` -- a no-op on a string
holding no colon -- handed the tag over as the repository, and promotion
rewrites exactly those tags.
Set an empty repository for a ref with no `/` ahead of its digest, so it
compares on the digest, which is the whole container identity such a ref
carries and the only part promotion must not move. Dropping these refs
instead would be wrong rather than blunter: packages/system/kuberture
carries an `image:` map with a `tag:` and no `repository:`, so shape 1
is the only rule that sees its digest, and skipping it would silently
stop proving that digest unchanged.
Reproduced over the release-1.6 packages tree with the shape-3 tags
rewritten rc-to-stable: before, the six pairs above diverge; after, both
sides normalize to 60 identical entries, kuberture's digest included and
kube-ovn's real repository still visible through its shape-4 join.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages_test.bats is byte-identical to main: 13
tests driving the verifier against a mocked `flux`, so they need no
registry. They cover the fail-open shapes that matter most — an
unreadable file during the rc-reference scan, a collector that returns
nothing, a digest that moved while candidate and merge tree still match.
hack/promote-gate-contract.bats is NOT main's copy, and the difference is
deliberate. main's version pins a `parse` job, a `skip_e2e_gate` input,
promote-time changelog and website-docs jobs, and the `labeled`
pull_request trigger they depend on — none of which exist here, because
`Promote RC` is a workflow_dispatch and runs from the ref it is
dispatched from, which is main. Run verbatim against this branch it fails
on its first test and the runner aborts the suite.
So this pins the half release-1.6 actually owns — the producer side:
* an rc tag push calls e2e-tag.yaml with the pushed ref, only for
-rc. tags, and grants the static permission ceiling the called
workflow declares;
* the called job's name is the exact literal main's gate composes and
searches for, asserted with grep -F because `${{` as a basic regular
expression matches nothing and would make the pin pass by finding its
own subject absent;
* the suite it runs is the full one (empty CHAINSAW_SUITES), with the
Makefile target and the sandbox chainsaw binary it needs;
* all five files promote-rc.yaml's preflight reads off the target base
are present, plus the two libraries the verifier sources by relative
path;
* verify-release-candidate is keyed on author and head branch and
carries NO label condition, so restoring main's from muscle memory
switches the job off here and fails this test instead;
* the candidate verification precedes the stable tag, the submodule
tag, the release publish, the retag and the chart publish, and
follows the toolchain, the login and the tooling checkout it needs;
* the tag-time changelog is validated and ported, not regenerated.
Every pin was mutation-checked: breaking the thing it describes turns it
red.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Since the promote PR is based on release-1.6, the changelog it carries
merges onto the maintenance line and never reaches main. The backstop job
in tags.yaml only asks whether the file is on origin/main, so it sees
"absent", spends an AI run, and produces text nobody reviewed — which
update-releasenotes.yaml then pushes into the published release body,
overwriting the reviewed notes the release actually shipped with.
Ask the second question too. `at_tag` is whether the changelog is present
in this checkout, which is the tag commit and therefore the promote merge
commit. When it is, port that reviewed file to main verbatim and skip
generation entirely; the AI now runs only when the changelog is genuinely
absent on both sides.
Add hack/validate-changelog.sh, byte-identical to main, and gate the
commit on it. A ported file came from the tag and a generated one came
from a step that is deliberately allowed to fail, so neither had been
checked: the old guard only asked whether the file was non-empty, which a
truncated Copilot stream satisfies. The validator asserts the H1, the
release-link comment, the trailing compare link and at least one section,
all carrying the expected version, so a fragment or a changelog generated
against the wrong tag cannot reach a release body.
Verified against this branch's own shipped changelogs: v1.6.1, v1.5.2 and
v1.5.1 all pass, and v1.6.1 checked against 1.6.2 fails as it should.
This takes effect for tags cut after it lands — tags.yaml and the script
it calls both come from the tag's own tree.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promote RC refuses to promote a release whose target base cannot verify
the packages candidate. It checks five markers on the base branch before
its first registry write and fails closed with "Target base
'release-1.6' lacks '<path>'". This branch satisfied one of the five
(hack/lib/image-refs.sh), so the next patch release off release-1.6
could not be dispatched at all.
Add the two halves the base has to carry:
* hack/verify-promoted-packages.sh and hack/lib/promoted-packages.sh,
byte-identical to main. The verifier pulls the candidate by its
committed digest and proves it matches the merge tree apart from the
artifact's impossible self-reference, that promotion rewrote every
version-line image tag, and that the container repo@digest set is
unchanged from the rc artifact.
* the two jobs that run it. verify-release-candidate in
pull-requests.yaml checks GitHub's prospective merge tree so a
packages/ edit pushed to the promote PR shows up as a failed check,
and a Verify stable packages candidate step in the finalize job of
pull-requests-release.yaml repeats it against the real merge commit.
Position is the whole point of the finalize half: it runs before the
write-once stable tag, the draft-release publish and the retag, so a bad
candidate is refused while nothing stable-named exists yet. The former
"Set up toolchain (skopeo, yq, helm)" and "Login to registry (GHCR)"
steps moved up with it and are no longer duplicated later in the job;
the toolchain now also installs flux and pins yq by version and sha256,
because an irreversible step must not depend on whatever
releases/latest/download resolves to today.
verify-release-candidate is keyed on the promote PR's author and its
head-branch shape rather than on the `release` label main matches. This
branch's on.pull_request.types has no `labeled` event, and the label is
applied about a second after the PR opens (a separate call in
gh pr create), so the `opened` payload carries no labels and a
label-keyed guard would never fire here.
.release-tooling is gitignored, matching main, so the trusted-base
checkout can never be staged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.
The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.
Three deliberate choices:
* The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
per-release and rc staging branches promote-rc.yaml and tags.yaml create
(release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
images come from the tag build, and rebuilding them would be waste.
* WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
image and build-main.yaml is deliberately its only, serialized writer so
concurrent builds cannot race on the cache manifest (the 409 class #2711
fixed for image tags). A line build can overlap a main build, so writing here
would reintroduce that race. Line builds read the cache.
* A missing artifact still degrades to committed refs, but on a release line it
now emits a ::warning:: naming the branch. Silent degradation is
indistinguishable from a working overlay, which is how a mis-specified branch
filter would hide for a whole release cycle.
Cost: one `make build` per push to a maintained line, i.e. per merged backport.
hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 2330d6f3ca)
The PR finalize job pulls cozystack-packages:main and repoints every package
the PR did not rebuild at current-main images. On a main-based PR that is the
whole point: it stops e2e from testing last-release images for everything
outside the build matrix. On a release-line PR it is wrong — the committed refs
there are not stale, they are that line`s released digests, and the charts are
written against exactly those.
So a release-line PR was installing main`s binaries onto its own charts, and the
mismatch grows with every commit main gains. #3437 is the demonstration: a
one-line change on release-1.6 that deactivates an app failed install
deterministically, twice, with
SchemaError(...core/v1alpha1.Option.spec): unknown model in reference:
"...core~1v1alpha1.OptionSpec"
from main`s cozystack-controller serving an aggregated OpenAPI that branch`s
charts cannot validate against. Nothing in the PR was broken; the lane was.
Left alone this makes every 1.6 backport look red, which is when release-branch
PRs are busiest.
Both overlay steps are now gated on `github.base_ref == main`, so a release-line
PR keeps its committed digests — the behaviour that predates the overlay.
Retargeting the overlay at a per-line artifact would be better but is not
possible today: build-main.yaml publishes only cozystack-packages:main, and the
registry carries no release-* equivalent (verified against the packages repo`s
tag list: `main` plus per-PR tags, nothing else).
hack/overlay-main-images_test.bats pins the wiring per step, so adding a third
overlay step without the guard fails the suite. Mutation-checked by removing one
guard. 12/12 green; actionlint clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit fea40d6314)
Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.
The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.
Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.
The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.
Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.
The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.
The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.
29/29 tests green; sh -n clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit e02adad38f)
Yan Bondarenko