A restarted capi-operator v0.19.0 drops the create events for the
provider Deployments it finds already running
(kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready
condition went False while the operator was down therefore keeps it
until that Deployment changes or the informer resyncs, and the first
resync after a restart lands two periods later because the handler
registers after the informer has listed. With the 10m default that is
about 20 minutes, during which capi-providers-core times out on the
CoreProvider and everything depending on it waits; every further
operator restart starts the clock again.
With a one-minute period the same repro recovered 2m02s after the
operator took its lease, against no recovery within 4m26s with the
default on the same cluster. A resync replays the cache and, with
providers in sync, ends in "No changes detected" and an unchanged
condition, so it writes nothing to the API server; the operator logged
about 15 lines a minute and used 2m CPU and 26Mi of memory.
The upstream chart has no value for the flag, so a package patch that
make update reapplies adds one, and a bats test holds the rendered
argument.
The test is adapted for the release-1.6 runner: hack/test_helper.bash
does not exist here, so the `load test_helper` line is dropped, and the
scratch file falls back to the cozytest.sh workspace. The assertion is
unchanged.
Assisted-by: LLM
(cherry picked from commit c2dab12a2e)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
#2946 added a startupProbe to capi-controller-manager so it survives the
certificate provisioning delay. It edited
capi-providers-core/files/core-components.yaml, which is the copy a reviewer
reads. The chart mounts files/components.gz, and that blob was never
regenerated, so the probe never reached a cluster.
The blob is regenerated from the YAML it is supposed to carry:
gzip -9 -n -c files/core-components.yaml > files/components.gz
The decompressed blob now matches the YAML byte for byte, and the only change
it carries is the six lines of #2946.
This reaches new installs only. capi-operator re-applies a provider when the
hash of its spec and its config Secret changes; the ConfigMap holding the
components is in neither, so a cluster that already runs capi-providers-core
keeps its manager without the probe until the provider's version moves.
Whether existing clusters should get the probe, and how, is left open here on
purpose: moving spec.version and the ConfigMap name together would force the
re-apply, but that is a version decision for the package, not a side effect of
regenerating a blob.
.helmignore is corrected in the same change: `files/.*-components.yaml` is a
glob, so it matched only names beginning with a dot, and `helm package`
shipped the readable YAML alongside the blob that duplicates it.
hack/capi-core-components_test.bats holds the two files to the same bytes from
now on, so the next edit of one without the other fails in CI.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
(cherry picked from commit 9934014908)
The nginx config baked into the console image kept the nginx defaults for
client_header_buffer_size (1k) and large_client_header_buffers (4 8k). An
OIDC access token carrying many group claims arrives as a single Cookie
header of several kilobytes, which no longer fits into one buffer, so
nginx answered 400 "request header or cookie too large" before the
request reached the /apis, /api or /k8s/ upstream.
Raise both to 16k, at the top level of that config rather than inside the
server block: a header field read before the Host header has selected the
virtual server is measured against the default server's buffers, so the
same lines inside a server block cover that server only while it stays
the default one. Measured against the pinned nginx base image, a 12 KB
cookie gets 400 on the old config and 200 on the new one. The ceiling
moves rather than disappears: a 20 KB cookie is still rejected.
This is the console hop alone. The platform ingress-nginx controller in
front of it runs on the same defaults, so where the dashboard is published
through an Ingress a token large enough to exceed them is still rejected
at the edge; where it is published through the gateway, whose edge has no
per-header ceiling of this kind, the console was the hop returning the
400. Raising the controller is a separate change in a separate package.
The accompanying check extracts the config from the Containerfile heredoc
and fails if either directive is missing, below 16k, or no longer ahead of
the first server block: the config lives inside an image build, where no
chart template and no helm-unittest case can reach it.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 7d53f7af22)
On the kube-ovn variants Cilium allocates no pod addresses, but the agent
still takes its router IP and the Gateway API Ingress IP from its
allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which
the documented installs slice from the same range kube-ovn allocates pods
from, so kube-ovn sooner or later hands a pod an address Cilium already
holds.
Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a
/29 per node from the last /21 of networking.podCIDR, and kube-ovn
excludes that /21 through --default-exclude-ips. The block stays inside
the pod CIDR on purpose. Envoy reaches Gateway API backends from the
node's Ingress IP, and kube-ovn masquerades host-originated traffic only
when the source lies in one of its subnets, so a block outside them such
as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes.
A post-upgrade hook moves existing clusters. It first checks that the
block has a /29 for every node and lies inside kube-ovn's default
subnet, and stops without changing anything otherwise. It adds the block
and every node's current Ingress IP to excludeIps of that subnet,
because the agent keeps a former Ingress IP in its ipcache as
reserved:ingress. It then waits until kube-ovn stops offering the block
and stops if an address in it is still allocated. kube-ovn leaves
excluded addresses out of the subnet status, so that check reads its IP
objects. Then, one node at a time, it clears the inherited /24 from the
CiliumNode, waits for the operator to allocate from the block and
restarts the agent.
networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn
variants, and the render fails otherwise. A /21 cut into /29s also caps
these variants at 256 nodes.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit dc362fa591)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Backport of #4828 to `release-1.6`.
PR lane on this line runs on Talos containers since the backport of
#4020, but `e2e-tag.yaml` (the rc gate that `tags.yaml` calls for every
rc from `release-1.6`) and `nightly.yaml` still boot QEMU guests. Every
rc tag run here since v1.6.2-rc.1 is red. v1.6.4-rc.1 failed with
kubernetes-latest and kubernetes-previous spending about 32 minutes each
at node-join deadline
(https://github.com/cozystack/cozystack/actions/runs/36124680874), while
the same 41 suites passed on container lane in the v1.6.4 promotion PR,
in a 93 minute job
(https://github.com/cozystack/cozystack/actions/runs/36383277279).
e2e plumbing on `release-1.6` diverged from `main` after #4020, so the
cherry-pick of the first commit has conflicts resolved against this
branch, and there is one prerequisite commit:
- `e2e-tag.yaml` and `nightly.yaml` get the same host module step,
`prepare-env-container`, endpoint and storage class env and two-part
teardown as `pull-requests.yaml` here. `e2e-tag.yaml` still requires the
release to carry its nocloud disk but doesn't download it anymore, and a
dispatch against a tag cut before container lane (v1.6.3 and older)
fails early and says to dispatch with `--ref <tag>`.
- nightly moves to 32-vCPU runner, because `hack/e2e-container-up.sh`
refuses a 24-vCPU host (`main` did the same earlier in db2ddfab2).
Tests: `platform-packages_test.bats` checks container lane env on all
three lanes, `promote-gate-contract.bats` covers the new guard (on
`main` it lives in `collect-images_test.bats`, there is no such file
here).
Differences from `main`: every lane keeps `timeout-minutes: 180`, the
follow-up comment fix about 215-minute budget is not included (this
branch has no such comment and no such value), fork lane wording is
dropped because there is no fork split here, docs edits cover only
paragraphs that exist on this branch.
Coverage lost on this line: DRBD and the `replicated` StorageClass,
Cozystack Talos node image with its extensions, CDI stock 600M worker
memory limit. Docs and comments that pointed at the rc run as that
coverage now say so. `nightly.yaml` runs only from `main`, so the change
there is for parity.
`tags.yaml` takes `e2e-tag.yaml` from the commit of the rc tag, so this
has to merge before v1.6.5-rc.1 is cut. If rc is cut first, dispatching
`e2e-tag.yaml` from `release-1.6` with that tag gives the same evidence.
`make bats-unit-tests` passes locally (487 ok, base has 486), but the
workflows themselves were not run yet. This stays a draft until
`e2e-tag.yaml` dispatched from this branch against v1.6.4-rc.1 is green,
I will put the run link here.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
ci(e2e): on release-1.6, the release-tag e2e gate and nightly run on Talos containers instead of QEMU guests. DRBD, the replicated StorageClass and the Cozystack Talos node image are no longer covered by any e2e lane on this line.
```
## What this PR does
Backport of the LINSTOR severity fix and its alert-rule guard from #3799
to `release-1.6`, together with the guard's follow-up #4812. Seven
piraeus-datastore rules use `warn`, which the default Alerta alarm model
does not accept, and they now use `warning`. Alert names, expressions
and thresholds stay unchanged.
Cherry-picked `3ade16a380c9d03b83ff28ea02af3b800fdd1b0f` and
`bd2cc0a749531433a237a9c8a6e21a6f21dee200` with their authors and
cherry-pick references. The KEDA severity commit from the same main PR
is left out because the KEDA rule it fixes does not exist on
release-1.6. The follow-up from #4812 is cherry-picked too. It makes the
guard catch a quoted kind or a kind followed by a comment, and fail on a
rule group it cannot see into once template actions are stripped.
### Validation
- All nine checks in `hack/alert-severity-contract.bats` pass with
`hack/cozytest.sh` on this branch. The wider discovery finds the same 39
rule files as before and no rule group comes out empty.
- Against `release-1.6` at `da98ddc08` without the fix, the same
contract fails on exactly the seven `warn` rules corrected here.
- Delivery through Alertmanager and Alerta was not exercised.
### Screenshots
Not a UI change.
### Downstream repositories
The file-level trigger map was checked against both changed files: a
rule file under `packages/system/piraeus-operator/alerts/` and a new
test under `hack/`. No existing tool, make target, API schema, package,
output resource or documented workflow is renamed or changed. No
companion repository update is required.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(linstor): backport accepted warning severity for seven piraeus-datastore alerts to release-1.6, with a regression guard for first-party alert rules
```
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.
Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.
DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit da28ea7ee8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The guard found rule files by an exact `kind: PrometheusRule` or
`kind: VMRule` line, so a quoted kind or one followed by a comment took the
file out of the check. A rule group whose rules all come from template
actions is empty once those are stripped, so it passed with nothing
checked. Discovery now accepts those spellings, and a rule document without
groups, or a group without a visible alert or recording rule, fails the run
the way a file that cannot be parsed does.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit dc814f3ae5)
Alerta refuses an alert whose severity is not in its alarm model, so a
rule with such a label is dropped every time it fires and the alert never
reaches anyone. Seven piraeus-datastore rules used warn and the keda
at-max rule used info, and nothing in the tree caught either.
The contract collects every PrometheusRule and VMRule document under
packages, in alerts files and in chart templates, and fails on any
alerting rule whose labels.severity is missing or not in the accepted
list. Helm actions are stripped before parsing, so a templated rule is
still checked, and a file that cannot be parsed fails the run instead of
being skipped. Values are compared as whole words.
none stays out of the list because the Alertmanager config routes it to
a blackhole receiver. Vendored charts under packages/**/charts/** are out
of scope because they are re-vendored rather than edited.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
(cherry picked from commit bd2cc0a749)
Migration 50 scales the legacy etcd-operator to 0, adopts every legacy
etcd.aenix.io cluster with etcd-migrate and then scales the operator
back to 1. At that point it is still the legacy operator, and it has
nothing left to manage. Until a tenant's etcd HelmRelease reconciles the
new chart, Flux can upgrade it with the old one, which re-creates the
legacy EtcdCluster that etcd-migrate just orphan-deleted. The legacy
operator then reconciles it, re-creates the legacy StatefulSet and
rewrites the legacy etcd-headless Service, dropping the owner references
etcd-migrate pointed at the adopted EtcdMembers. When the new chart
prunes the legacy EtcdCluster, the garbage collector deletes the Service
with it, and the adopted members' etcd-<i>.etcd-headless names stop
resolving.
After a successful adoption the operator now stays at 0 until the
etcd-operator HelmRelease replaces the Deployment with the v1alpha2
operator later in the same upgrade. A failed or interrupted adoption
still restores it through the EXIT trap.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 6248d458ad)
# Description
Backport of #4253 to `release-1.6`, together with #3460 which it depends
on. On 1.6 kube-ovn chart and image still come from
cozystack/kubeovn-chart, so draft Dockerfile had nothing to build it and
kubeovn build failed (no `image` target). #3460 moves build in-tree on
same kube-ovn 1.15.10 this branch runs, and brings nightly-mirror fix
for kube-ovn split registry host that becomes needed once kube-ovn is
built by CI. Bot draft also dropped second commit of #4253
(`fix-vmim-target-pod-retry.diff`), it is back now.
I built image locally, all three patches apply to kube-ovn v1.15.10 and
`TestVMIMigrationSchedulingRetry` passes.
The CAPK build suite came over from main asserting that an image build
under OCI_EXPORT_DIR writes an OCI archive with --push=0. That export
path lives in hack/common-envs.mk on main and was never brought to this
branch, where every build pushes, so the suite failed here on every
run.
What the suite is for still holds on this branch: the build stamps a
digest-pinned ref into the .tag file that the image-ref tooling finds,
it leaves the shipped components untouched, and a failed build or a
missing digest keeps the previous pin. Those checks stay; the OCI
parameter and the three archive checks go.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/bats-no-exit-trap.bats is an exact ratchet, and the new suite installs
three traps inside explicit subshells without declaring them, so
make unit-tests fails on the merge commit. The header now declares the
count and says why the three are not debt.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 0e4d6a305c)
The nightly host rewrite is a literal "<src-registry>/" substring
replace, so it only reaches a host sitting contiguously in front of the
repository. kubeovn keeps its host in global.registry.address with the
bare repository in a sibling key: no trailing slash, no match. The
image was mirrored to the public registry and the tree kept pointing at
the private build registry.
The gap was dormant while kubeovn's image came from a separate
repository, because the baked ref carried the public host and the
ownership filter skipped it. Building the image in-tree stamps the
build-registry host into that key and turns it live, so fix it here.
Add an end-of-line-anchored expression that fires only when the source
registry is the complete value; a contiguous ref continues past the
host, so the two expressions cannot both fire on one reference.
keycloak-operator splits at a different boundary and stays unfixed --
neither of its keys holds the source registry as a whole. Documented,
and dormant while no CI path rebuilds it.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 6cb579338b)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Both suites matched "strimzi.io/cluster=kafka-test" as a bare substring,
and that is a prefix of the sibling release kafka-test2 and of
kafka-test-topics. Rendering the hook with either selector left both
suites green, so neither pinned the scoping they exist to pin. Anchor on
the leading "-l " and on the trailing delimiter; both variants now red in
both suites, and the six earlier hook mutations still red exactly where
they did.
Drop the notMatchRegex on a leading slash. It cannot fail on its own: the
anchored image pattern above rejects a leading slash first, and an empty
_cluster.images-registry renders no slash to begin with. Its rationale
moves into that pattern's comment.
Record why activeDeadlineSeconds and the 180s wait have to move together.
A deadline that fires mid-wait kills the Job, and with backoffLimit: 0 the
hook then returns non-zero, which is the wedged uninstall the deliberate
exit 0 on the timeout branch exists to avoid.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 637098f222)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The suite used bats' run helper. make bats-unit-tests does not invoke the bats
binary: it runs hack/cozytest.sh over every hack/*.bats, and that runner
implements @test and little else, so the file died at 'run: command not found'
and took make unit-tests with it. Under bats it was 5/5, which is where the 5/5
in the previous commit came from. Save $? by hand instead, the shape
app-cleanup-hooks_test.bats already uses; verified under both runners.
The fake kubectl no longer defaults DELETE_RC to 0. An unset value made every
failure-path assertion vacuous; now it exits 64 and says so. Cover both
RESTMapper wordings for an absent CRD rather than just the first.
Widen the image pattern to v[0-9]+\.[0-9]+(\.[0-9]+)? like harbor and mariadb:
pinning v1\.32 reintroduced the breakage the digest was made pattern-based to
avoid, one Renovate bump later. Assert the _cluster.images-registry override
and the no-leading-slash default, neither of which was covered: the hook is the
one image in this chart whose pull failure surfaces at uninstall.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 415aa82409)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The hook's exit code is the whole contract: a non-zero one returns from
Uninstall.Run before any release resource goes, keeping the topic operator
alive; a zero one lets Helm remove it. delete_test.yaml matches manifest text,
so it stayed green when the timeout branch was put back to exit 1, when the
CRD-absent branch was, and when the selector was widened past this release.
Add a bats suite in the shape hack/app-cleanup-hooks_test.bats already uses for
qdrant and harbor: render the script, run it against a fake kubectl, pin the
exit code of each branch. Each of the three mutations above now turns it red.
Drop hook-succeeded from the delete policy and reclaim the Job with
ttlSecondsAfterFinished instead. The script exits 0 on the degraded path too,
where topics still hold the finalizer, so deleting on success removed the only
record of that seconds after it was written. Assert both in delete_test.yaml.
Pin the hook image by shape rather than by literal digest: Renovate rewrites
digests under packages/*/templates/hooks/ and nothing rewrites tests/, so the
literal turns the next bump red for no defect.
Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 3326e0fb9e)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The post-delete absence assertions could pass vacuously if the two
release topics were never created. Asserting both topics first makes
the E2E prove that the pre-delete cleanup actually removed them.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
(cherry picked from commit 9d2fddd4e6)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The vminstance suite waits for each VMDisk's DataVolume to be Ready
before anything consumes it. On `replicated` that works because the
class binds Immediate; on the container lane's `local` class, which is
WaitForFirstConsumer, CDI starts no importer for a disk nothing has
consumed yet, so the DataVolume sits in PendingPopulation and the suite
times out with no error anywhere. main makes the vm-disk chart request
immediate binding for every source; the chart on this branch requests it
only for uploads, and the lane must not change what it ships.
So on the container lane, right after each VMDisk is applied, the suite
waits up to 180s for CDI to create the DataVolume and its claim and
annotates both with cdi.kubevirt.io/storage.bind.immediate.requested,
the same request the chart already writes for uploads. Both, because
the DataVolume decides what phase CDI reports and the claim is what the
populator reads to start the import without a consumer. The QEMU lane
annotates nothing.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
CDI's own 600M worker memory limit was seen to OOM the decompress and
convert of a tenant worker disk near 100% on the container lane, after
which CDI retries from scratch and the DataVolume cycles forever rather
than failing. The same import completes at 600M on QEMU nodes, so what
needs the headroom is the substrate. main passes the raise through a
kubevirt-cdi chart value that this branch does not have.
The last install test now merges the same figures into the CDI CR on
the container lane: 4Gi and 256Mi for memory, CPU left at CDI's 750m and
100m because the failure was memory. It has to be the CR, not CDIConfig,
because the operator reconciles CDIConfig.spec from the CR and reverts a
direct edit, and the test waits for CDIConfig's published default to
read 4Gi, since that is what CDI hands its worker pods. The QEMU lane
keeps CDI's defaults.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
On the container lane every volume is on `local`, which is node-pinned
and WaitForFirstConsumer. Without --strict-topology external-provisioner
passes every topology segment as requisite and only prefers the node the
scheduler picked, so linstor-csi may provision elsewhere when that node
is full. A CDI importer mounts two such volumes and the scratch one is
created only after the pod is scheduled, so it can land on another node
and leave the importer unschedulable forever with nothing erroring.
main turns the flag on in the linstor chart; the chart on this branch
cannot, and the lane must not change what it ships.
So the last install test, on the container lane only, adds the same
csi-provisioner override to the live LinstorCluster, waits for
piraeus-operator to carry it into linstor-csi-controller and for that
rollout to finish, and asserts the flag on the Deployment. `args` has no
patchMergeKey, so the list is restated in full as piraeus-operator
v2.10.2 renders it. It runs last because the earlier tests that change
platform values upgrade linstor, and an upgrade re-renders the
LinstorCluster without the override. For the same reason the tenant
Kubernetes suites and both vminstance Tests re-check the flag before
they import, so an override lost to a later upgrade fails by name rather
than as an import stuck at ImportScheduled.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
The linstor chart adds a drbd-logger sidecar to every satellite, and on
the container lane's shared kernel there is no DRBD for it to tail, so
it exits at once. The satellite pod is then never Ready, piraeus never
registers the node, LINSTOR reports no nodes and no PVC binds. main
drops the sidecar through a chart value; the chart on this branch has
none, and the lane must not change what the chart ships.
So post-install prep applies a LinstorSatelliteConfiguration of its own
on the container lane, ahead of the linstor waits, with the same
DaemonSet patch main renders: delete drbd-logger from the generated
satellite DaemonSet. The two DRBD-only initContainers are already gone
through the chart's Talos satellite configuration. Once all three nodes
are Online, prep reads the live DaemonSets back and fails if a
satellite still carries the sidecar or none was found at all, so a
configuration that silently stopped applying cannot pass for one that
worked. The QEMU lane applies nothing and keeps the chart as shipped.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
The three checks that System-mode artifacts are gone after the mode
switch were `if kubectl get ... >/dev/null 2>&1`, which reads every
non-zero exit as "the object is gone". An RBAC denial, a timeout or a
missing CRD produces the same exit, and stderr went to /dev/null, so the
suite reported a clean System teardown for never having observed one.
--ignore-not-found is what separates absent from unaskable: absent is
exit 0 with empty output, everything else stays non-zero. Preferred over
matching NotFound in the message, which depends on wording and, under a
runner with tracing on, captures the trace instead of the error.
Each probe is pinned individually rather than as a group: the new case
breaks one query per iteration while the other two answer absent, so
reverting any single probe to the bare form reddens the suite. Verified
by doing exactly that three times. A test that broke all three at once
would have passed on any one surviving probe, which is how the first
attempt at this test fooled me.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit b30c7efaf6)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
.chainsaw.yaml states the rule and says why: an op that expires before
its inner bounds do is a SIGKILL on the process group, which takes out
the diagnostics the failure branch exists to print, while an inner
timeout leaves a partial capture and lets the remaining steps run.
refresh-and-assert-full-health, added on this branch, breaks it. The
poll deadline is 270s, the last in-loop read can start just under it and
run 25s, and the failure branch then reads again for another 25s: 320s
of inner bounds under a 5m op. Raised to 6m.
verify-processes-and-security-context breaks it harder and did so
before this branch: 240s for the process counts, then 120s for
connectionString and 120s for generations.reconciled, sequentially, is
480s under the same 5m. Fixed here rather than filed, because it is the
same defect one step away, found by applying the rule to the neighbour.
Raised to 9m.
The kubernetes-latest ceiling gets the measurement instead of a change.
The OIDC lifecycle folded onto that cluster adds up to 1m + 600s + 1m of
passing-path waits that the 67m op does not price, and raising the op is
not available: a guard holds COZY_OP_CEILING equal to both suites'
timeout, and two suites at 79m plus their teardowns exceed the job's
215-minute cap. On the green run of this branch the whole Test took
569.72s against the 4020s op, with kubernetes-previous at 470.19s. The
ceilings only bind once something is already hung, which is what the
existing residual note (#3666) says; now it says it with a number.
Adapted for release-1.6: the kubernetes-latest op here is 40m and no
guard ties it to another figure, so its note states what it does not
price instead of citing the COZY_OP_CEILING guard, the 67m ceiling and
the main-branch measurement, none of which exist on this branch.
Assisted-By: LLM
(cherry picked from commit b0e0f38fbe)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The readiness gate resets its stability window when a list read fails,
and nothing held that: the existing "transient list error" case fails on
the first read, when there is no window to invalidate, so removing the
reset left the suite green. The new case breaks a read mid-window, where
a gate that kept counting through the gap would accept a release set it
never observed stable across it.
The parallel fan-out contract asked make for --output-sync, which arrived
in GNU Make 4.0 while macOS still ships 3.81 -- so on a contributor's
machine the suite died on an unknown option instead of reporting the
contract. Guarded by version, with the reason in place. CI runs 4.x, so
the contract is still enforced where it gates a merge.
Adapted for release-1.6: the --output-sync guard is dropped together
with hack/unit-test-parallelism.bats, which this branch does not have
because the unit-test fan-out it covers is not backported.
Assisted-By: LLM
(cherry picked from commit 8456facc79)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two ways the gate did not do what its arguments promise.
The list read carried no bound. kubectl defaults to no request timeout,
and the read sits above the deadline test at the bottom of the loop, so
against a wedged apiserver it blocks there and timeout_seconds stops
being a ceiling: what arrives is the job timeout as a kill, instead of
the not_ready dump this function exists to print. Bounded with kubectl's
own --request-timeout rather than a `timeout` wrapper, because the unit
suite substitutes a kubectl shell function and `timeout` execs a binary.
The final diagnostic read is bounded the same way.
`now` was sampled before the read, so a read costing more than the poll
interval was priced at zero against the deadline and bought the loop
another whole iteration past it. Sampled after, on both branches.
And the count >= minimum_count half of the gate was pinned by nothing:
every fixture returned 11 or 12 items against a minimum of 11, so the
condition could be deleted outright and all five tests stayed green.
That threshold is the direct replacement for the `wc -l` existence
backstop this branch removes from hack/e2e-install-cozystack.bats -- the
only thing separating "everything is Ready" from "almost nothing got
created", since three HelmReleases that exist are all Ready. Both new
cases were checked by reverting the code they cover.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit ec7f17299b)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The srv1-srv3 nodes of both lanes that gate a merge -- the same-repo one
in pull-requests.yaml and the fork one in e2e-fork.yaml -- run as Talos
containers instead of QEMU guests, so a tenant worker sits at L2 rather
than L3. Measured in CI on commits where both substrates ran the same
tree: whole-job wall clock down 24-49 minutes, essentially all of it in
Chainsaw, while the two tenant-Kubernetes suites go from 1143-2040s to
466-809s and stop depending on the runner's vgif flag.
The node-join soft-red gate goes with the substrate, which is what it
was for. It existed because nested virtualisation on the shared runners
degraded far enough that a worker registering in minutes elsewhere could
miss any deadline this test could afford; #3513 established that the
discriminator was the host's kvm_amd vgif flag, and removing a nesting
level removes what it acted on. So hack/e2e-node-join-soft-red.sh, the
SOFT-RED-node-join.txt marker and the soft_red job outputs are gone from
all four workflows, and a missed deadline is an ordinary red again. The
::warning annotation survives, because the reason a run went red should
be legible without opening the log, and so does the 124-versus-everything
-else test, because "the workers were slow" and "the wait never ran"
send a reader to different places.
Done by swapping the substrate inside the existing e2e jobs rather than
by promoting a separate container job. Those jobs carry more than a
substrate -- the GitHub App token, the report-overrun guard, the images
list, the SSH breakpoint -- and promoting a job that never had them
would have dropped four features without saying so.
Three properties of container mode drive the design, and each fails
quietly rather than loudly. machine.kernel.modules is a silent no-op --
kernel_module_spec.go returns early on ModeContainer with no error and
no event -- so the host loads openvswitch and zfs before the nodes start
and the workflow step asserts both rather than trusting them; a missing
module surfaces an hour later looking like a CNI or a storage
regression. Docker gives a privileged container its own tmpfs /dev
seeded once at creation, so /dev/kvm works but ZFS zvols created later
never appear and linstor-csi fails ControllerPublishVolume; the compose
file binds the host devtmpfs, which is also why talosctl cluster create
docker can never carry LINSTOR. And there is no ARP VIP, so
192.168.123.11 stands in for the QEMU lane's .10.
The QEMU machinery stays in the tree and keeps running: nightly.yaml and
e2e-tag.yaml still call `make prepare-env`, so DRBD, the replicated
StorageClass, its Immediate binding mode, the tenant StorageClass
propagation that only applies to remotely-accessible classes, and the
Cozystack Talos node image with its extensions all keep nightly and
release-candidate coverage. None of them is exercised per pull request
any more. That is the trade.
Nothing on the PR path builds or downloads a nocloud disk now, so
build-talos drops the talos-nocloud step and the talos-image artifact.
build-talos itself stays: finalize consumes its digest fragment.
One property this does not fix, recorded beside the runner class because
it will be read off a red run eventually: a container node reports the
HOST's CPU and memory as capacity, and hack/e2e-container-up.sh counters
that with kubelet systemReserved. That corrects the scheduler and not
what a pod reads from /proc, so a workload sizing itself from
/proc/cpuinfo or /proc/meminfo is configured for the whole runner.
Also carried here because the same files carry them: the ghcr.io
pull-through mirror removal (see #4007 -- whichever lands first makes the
other a no-op), the authoritative HelmRelease readiness gate, the
backup-credential preflight, the prepull guard against an image-less
container fragment, and folding the two standalone OIDC suites into the
latest-version tenant cluster.
Adapted for release-1.6: this branch has no e2e-fork.yaml, no soft-red
gate, no ghcr.io mirror and no backup e2e, so those hunks are dropped
together with the tests for helpers that exist only on main; e2e-tag.yaml
stays on QEMU. The platform Package generator carries the API endpoint
and nothing else, because the linstor drbd.enabled and kubevirt-cdi
importerResources overrides need chart values this branch does not ship;
the container lane gets e2e-side substitutes for them in later commits.
run-kubernetes.sh takes the storage-class helper, the local-lane pool
baseline, the scoped drain and the OIDC fold, while cleanup keeps its
best-effort `|| true`. The OIDC suite references in select-install.sh and
the chainsaw README go with the suites here rather than in a later commit.
(cherry picked from commit 186547d835)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-by: LLM
verify_storageclass_fallback_default deletes the `replicated` class,
renders with --dry-run=server, and restores it inline "before any
assertion exit" -- the comment's words. Under the caller's `set -eu` the
render is a plain assignment, so a failed render exits the function
there: before rc is read, and before the restore runs. The management
cluster is then left with no default StorageClass for every suite after
it, which is the opposite of what the comment promises.
Captured with `|| rc=$?`. Pre-existing shape, but the restore block is
edited on this branch anyway, and the QEMU lanes are where it bites.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
(cherry picked from commit b131bd56a4)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Backport of #3579 to `release-1.6`.
Tenant backend Deployment in the kubernetes suites still has one 300s
budget for scheduling and image pull on this branch. Right now those
suites on 1.6 mostly fail earlier, on node-join, so this starts to
matter once #3455 widens that window.
Cherry-picked clean with `-x`.
### Testing
- `make unit-tests` green, POSIX sh sweep clean.
```release-note
NONE
```
Backport of #3715 to `release-1.6`.
Both linstor waits in `hack/e2e-post-install-prep.sh` still run off one
15m deadline on this branch, so a slow linstor HelmRelease eats the
Deployment's budget and install reports the wrong link as broken. Each
wait now gets its own budget.
Cherry-picked clean with `-x`.
### Testing
- `make unit-tests` green, POSIX sh sweep clean.
```release-note
NONE
```
Backport of #3750 to `release-1.6`.
Sandbox Talos config and platform install on this branch still both use
`10.244.0.0/16`, so kube-ovn and cilium allocate from one range and
cilium refuses pod sandboxes with `IP ... is already in use`. It hit
`mariadb-2-backup-roundtrip` on the E2E run of #4328 here.
Cherry-picked clean with `-x`.
### Testing
- `make unit-tests` green, POSIX sh sweep clean.
- Real check is E2E on this PR.
```release-note
NONE
```
The tenant backend Deployment had a single 300s budget to reach
condition=Available, and two unrelated variable costs shared it. The
first is scheduling. What the suite establishes before that point is two
tenant nodes Ready, which is weaker than schedulable: a Ready node still
carries node.cilium.io/agent-not-ready until the tenant cilium agent
claims it, and a node the bringup has not finished with is
SchedulingDisabled. Scheduling took 2m18s and 1m57s in the two tenant
suites of one run, leaving the image pull to finish inside what was
left. It did not, and both suites failed with the same message for two
different shortfalls.
Wait for a node that actually accepts a toleration-free Pod before
creating the workload, on its own budget and its own failure message.
The gate encodes the scheduler's rule for such a Pod (Ready, not
unschedulable, no NoSchedule or NoExecute taint) over a custom-columns
probe, prints the node table on both outcomes so a timeout names the
taint that held it, and treats a failed probe as not-schedulable so an
API blip cannot release it. The readiness wait keeps its 300s, now
starting from a schedulable node, and dumps deployment, pod and event
state when it runs out.
Pin the workload image by digest. The tenant workers reach no registry
mirror, so nginx is pulled from Docker Hub on every run either way; the
digest fixes what that pull returns instead of leaving a floating tag
free to change size and content under a fixed deadline.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 5542ae52b7)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The post-install prep script waits out a serial reconcile chain --
the linstor HelmRelease, then the linstor-controller Deployment --
against a single 15-minute deadline anchored at the script's start.
The head of the chain therefore eats the window and the tail inherits
whatever remains, so an install fails whenever the chain merely runs
long rather than never converging.
Measured on one such failure: the HelmRelease went Ready 452s in, and
the Deployment needed a further ~520s because the controller's DB
migration lost its apiserver connection and retried four times before
succeeding. Neither figure exceeds 900s on its own; only their sum
did, and the install reported the Deployment as though it were broken.
Give each wait its own budget, started when the link before it
completed -- the shape the LINSTOR node and MetalLB CRD waits further
down already have. What that buys is not speed but a guarantee: every
link gets its whole allowance regardless of what the links ahead of
it spent, so the tail can no longer be failed by the head being slow.
It is a trade, and the other half is worth stating. The script's worst
case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job
carries, where before the two waits shared one 15m window. And a link
that is genuinely stuck is reported later rather than sooner: a
Deployment that never converges surfaces at 1355s where the shared
window surfaced it at 900s, on that run's timings. A HelmRelease that
never becomes Ready surfaces at the same moment as before, since its
budget still starts with the script.
A timeout now also lists the linstor pods and the namespace events.
The description names the object waited on, not the link that held it
up, and the same "Deployment not Available" has been reached both with
the controller pod crash-looping its migration init container and with
that pod blocked on a client TLS Secret cert-manager had not issued
yet. Telling those apart previously required the run's bundle.
The events come from `kubectl events` rather than `kubectl get
events`. It orders by when an event was last seen, so a condition
repeating for minutes -- what a stuck mount looks like -- stays inside
the window instead of being pushed out by newer one-off events, and it
carries no --sort-by, so it cannot fail the whole read the way a sort
key does when one item lacks it.
Both reads keep their stderr and carry a bound. A read that fails
silently is indistinguishable from a namespace with nothing to report,
which is the distinction being drawn; and the caller is blocked in
`wait`, so an unbounded read against a wedged apiserver would hold the
install open until the job's own timeout. The client budget is
strictly the smaller of the two so kubectl gets to name the reason
before the outer kill takes it away.
The wait helper is named for linstor because those diagnostics read
one namespace unconditionally: the wait is generic over its kubectl
arguments, what it dumps on the way out is not.
The new unit test runs the script end to end against a stub PATH
carrying a virtual clock, so a 15-minute budget elapses in a few
hundred forks and the timings are exact rather than wall-clock
approximate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 86674c7084)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The sandbox Talos config and the platform install step both used
10.244.0.0/16, one as the Kubernetes cluster CIDR and the other as the
kube-ovn POD_CIDR, so two allocators drew from one range.
kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium is
chained behind it and allocates no pod addresses, but it still carves
its own per-node router IP and Ingress IP out of Node.spec.podCIDR,
which kube-controller-manager slices from the cluster CIDR. Sooner or
later kube-ovn hands a pod an address cilium already holds. When that
address is the Ingress IP the agent refuses the sandbox with "IP
ipv4:<addr> is already in use", and cilium never relents on it, so the
pod only proceeds if something reschedules it onto a different address.
In the failure this comes from, the same address was refused seven
times across six pod UIDs before the suite ran out of budget. When it
is the router IP nothing complains at all and only host-to-pod traffic
breaks.
Move the cluster CIDR to 100.65.0.0/16, which no other range in the
tree uses. That deliberately stops the sandbox matching the layout the
install documentation prescribes, so it is not a value to put back.
Pod addresses are unaffected: they still come from kube-ovn out of
10.244.0.0/16, and cilium does not require a chained endpoint's address
to sit inside its own allocation CIDR.
Excluding the addresses on the kube-ovn side would not work, because
cilium picks its offsets by random scan over the node's range.
The new guard checks the invariant rather than the literals, so either
value may move as long as they stay disjoint. It covers the service
and join ranges too, since landing on one of those instead is the next
way to get this wrong.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
(cherry picked from commit 9ace63b441)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The kubernetes-latest and kubernetes-previous suites flaked on the
worker node-join wait under host load. In run 30260770694 both failed at
exactly 12m with zero tenant Nodes registered and the tenant cilium HR
still mid-install: a transient drbd.linbit.com/lost-quorum taint delayed
the worker DataVolume imports, so the KubeVirt VMIs were only created ~2m
into this wait and the guests then had ~10m to import Talos, boot,
register a kubelet and let cilium turn the nodes Ready.
This single deadline absorbs the entire worker bring-up (the
machinedeployment replicas=2 gate clears before the VMIs even exist), so
under storage pressure 12m is too tight. A less-loaded fleet run passed
both suites unchanged, confirming load-induced slowness rather than a
stuck bring-up. Widen the deadline to 18m, which restores margin and
still sits well inside the 40m chainsaw step timeout.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit 330acfe6e2)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The e2e QEMU guests boot with three raw virtio -drive lines that carry
no cache= option, so QEMU defaults to cache=writeback and honors every
guest fsync. The management-cluster Talos etcd fsyncs on each Raft
commit, while containerd unpacks dozens of images and CDI imports
multi-GB VM disks onto the same host-backed raw files during the
VM-heavy suites. The fsync storm serializes etcd's writes into
multi-second stalls, the mgmt apiserver times out lease PUTs, and many
operators (capk, etcd-operator, fdb-operator, cilium-operator, kamaji)
lose leadership at once — failing whichever chainsaw suite is
mid-flight, a different one each run.
Add cache=unsafe to each guest drive so guest flushes become no-ops
against the host page cache, removing the fsync-gated etcd stalls. This
is safe only because these CI VMs are ephemeral and destroyed every
run; never use it for real data.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
(cherry picked from commit ad35afa534)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review of the version generator (hack/update-versions.sh):
- Fetch tags with `skopeo list-tags` (registry v2 /tags/list) instead of
the Docker Hub v2 API, whose anonymous pagination caps at 1000 entries
and then returns an error page with no `.results`, silently truncating
the list so the frozen 24.9.2.42 fell outside the window and the run
failed. skopeo is already in build-deps and is how the sibling
generators (postgres, redis, mongodb, opensearch, kubernetes) fetch.
- Make a frozen default tag missing from the registry a hard error
instead of silently falling back to a newer patch, which would move
the default image for existing installs.
- Restore 0644 on the regenerated files (mktemp's 0600 survived the mv).
- Exclude /hack from the chart tarball via .helmignore.
- Refresh files/versions.yaml: v25.8 -> 25.8.32.4, the true latest 25.8
patch now visible with the complete tag list.
- bats: add a missing-frozen-tag hard-error case and make the atomicity
test reach the resolution phase after a valid default rather than
failing at the default-in-set check.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit e707475f96)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
Add a `version` value (v25.8, v25.3, v24.9; default v24.9) to the
ClickHouse chart, backed by files/versions.yaml and resolved by
templates/_versions.tpl for both the ClickHouse server and Keeper
images, so the Cozystack API returns the engine version (spec.version)
like the other managed-DB charts. Default v24.9 -> 24.9.2.42 keeps
existing installations byte-for-byte identical.
The maintainer-run hack/update-versions.sh generator intersects the
server/keeper Docker Hub tags byte-collated (LC_ALL=C) so comm keeps
tags across the X.9 -> X.10 boundary; freezes the default major to the
tag it already ships (read from versions.yaml, so the freeze follows the
default); errors on a dropped default or an unresolvable major; writes
atomically; and runs on stock macOS bash 3.2 / BSD. Covered by
hack/clickhouse-update-versions_test.bats (10 cases) and the
tests/version_test.yaml helm-unittest suite.
Refs: #1246
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 3c78c0be56)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 5bcdb8cfb8)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: LLM
No fixture in verify-promoted-packages_test.bats built a shape-3 split
map -- `repository:` on one key, `tag: <tag>@sha256:<digest>` on another
-- which hack/lib/image-refs.sh itself calls the dominant shape in the
tree. The existing fixture's single-string ref carries a registry host,
so its rc-to-stable rewrite normalizes through the other branch and
passes whether or not host-less refs are handled, which is why the suite
looked covered while a clean promotion could still be rejected.
Add two cases. The first promotes a split map whose tag moves from rc to
stable with the repository and digest held constant; reverting the
normalization fix reddens it on
"promotion changed the container repository/digest set", diffing
v9.9.9-rc.3@sha256:eee... against v9.9.9@sha256:eee... -- the same
tag-as-repository pair, with identical digests, that CI reported.
The second pins the property a blunter fix would break. A digest-only
`image:` map with no `repository:` (the kuberture shape) is seen by no
rule but shape 1, so it must still contribute its digest: the case moves
that digest and nothing else, and asserts the rejection comes from the
digest set rather than a neighbouring leg. Replacing the fix with an
outright `continue` over host-less refs makes the verifier exit 0 on that
changed digest and reddens the case at its exit-status assertion.
Negative assertions are counted rather than `!`-negated: under
hack/cozytest.sh a `! grep -q` cannot fail, so it would be a comment
shaped like an assertion.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages.sh reported "promotion changed the
container repository/digest set" for a v1.6.2 promotion that moved no
container bytes at all: all 48 real repo@digest refs and all 48 digests
were byte-identical between the rc artifact and the promotion candidate.
The six entries it flagged as changed repositories were not
repositories. They were bare tag strings -- v1.6.2-rc.1 on the rc side,
v1.6.2 on the candidate side -- pairing one-to-one by identical digest.
hack/lib/image-refs.sh emits two entries per shape-3 split map: the
correctly joined repository@digest, plus shape 1's recursive scrape of
the raw .tag scalar, which carries no repository. The library documents
that degeneracy and leaves host-less refs to its callers;
promote-retag.sh drops them, the verifier had only a
*/cozystack-packages exclusion. So `${image%:*}` -- a no-op on a string
holding no colon -- handed the tag over as the repository, and promotion
rewrites exactly those tags.
Set an empty repository for a ref with no `/` ahead of its digest, so it
compares on the digest, which is the whole container identity such a ref
carries and the only part promotion must not move. Dropping these refs
instead would be wrong rather than blunter: packages/system/kuberture
carries an `image:` map with a `tag:` and no `repository:`, so shape 1
is the only rule that sees its digest, and skipping it would silently
stop proving that digest unchanged.
Reproduced over the release-1.6 packages tree with the shape-3 tags
rewritten rc-to-stable: before, the six pairs above diverge; after, both
sides normalize to 60 identical entries, kuberture's digest included and
kube-ovn's real repository still visible through its shape-4 join.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages_test.bats is byte-identical to main: 13
tests driving the verifier against a mocked `flux`, so they need no
registry. They cover the fail-open shapes that matter most — an
unreadable file during the rc-reference scan, a collector that returns
nothing, a digest that moved while candidate and merge tree still match.
hack/promote-gate-contract.bats is NOT main's copy, and the difference is
deliberate. main's version pins a `parse` job, a `skip_e2e_gate` input,
promote-time changelog and website-docs jobs, and the `labeled`
pull_request trigger they depend on — none of which exist here, because
`Promote RC` is a workflow_dispatch and runs from the ref it is
dispatched from, which is main. Run verbatim against this branch it fails
on its first test and the runner aborts the suite.
So this pins the half release-1.6 actually owns — the producer side:
* an rc tag push calls e2e-tag.yaml with the pushed ref, only for
-rc. tags, and grants the static permission ceiling the called
workflow declares;
* the called job's name is the exact literal main's gate composes and
searches for, asserted with grep -F because `${{` as a basic regular
expression matches nothing and would make the pin pass by finding its
own subject absent;
* the suite it runs is the full one (empty CHAINSAW_SUITES), with the
Makefile target and the sandbox chainsaw binary it needs;
* all five files promote-rc.yaml's preflight reads off the target base
are present, plus the two libraries the verifier sources by relative
path;
* verify-release-candidate is keyed on author and head branch and
carries NO label condition, so restoring main's from muscle memory
switches the job off here and fails this test instead;
* the candidate verification precedes the stable tag, the submodule
tag, the release publish, the retag and the chart publish, and
follows the toolchain, the login and the tooling checkout it needs;
* the tag-time changelog is validated and ported, not regenerated.
Every pin was mutation-checked: breaking the thing it describes turns it
red.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Since the promote PR is based on release-1.6, the changelog it carries
merges onto the maintenance line and never reaches main. The backstop job
in tags.yaml only asks whether the file is on origin/main, so it sees
"absent", spends an AI run, and produces text nobody reviewed — which
update-releasenotes.yaml then pushes into the published release body,
overwriting the reviewed notes the release actually shipped with.
Ask the second question too. `at_tag` is whether the changelog is present
in this checkout, which is the tag commit and therefore the promote merge
commit. When it is, port that reviewed file to main verbatim and skip
generation entirely; the AI now runs only when the changelog is genuinely
absent on both sides.
Add hack/validate-changelog.sh, byte-identical to main, and gate the
commit on it. A ported file came from the tag and a generated one came
from a step that is deliberately allowed to fail, so neither had been
checked: the old guard only asked whether the file was non-empty, which a
truncated Copilot stream satisfies. The validator asserts the H1, the
release-link comment, the trailing compare link and at least one section,
all carrying the expected version, so a fragment or a changelog generated
against the wrong tag cannot reach a release body.
Verified against this branch's own shipped changelogs: v1.6.1, v1.5.2 and
v1.5.1 all pass, and v1.6.1 checked against 1.6.2 fails as it should.
This takes effect for tags cut after it lands — tags.yaml and the script
it calls both come from the tag's own tree.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil