Commit Graph
7777 Commits
Author SHA1 Message Date
Aleksei Sviridkin 200a7fff66 test(e2e): read the kafka round-trip cleanup order without yq if/else
The check on kafka-2-backup-roundtrip's finally block classified each
op with if/then/elif/else, which yq only gained in v4.54.1. The CI
runner's yq predates that and rejects the whole expression with a
lexer error, so the test failed there while passing locally.

Have yq emit the plain facts per op and let awk name it. The round-trip
document is cut out before that query, because in a multi-document file
yq also evaluates an array literal against the documents a select
drops and prints a line for each.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 02:49:36 +03:00
Aleksei Sviridkin cb7a579065 fix(e2e): compare LINSTOR pool datasets, not free bytes, at cleanup
For a node with only thick ZFS pools, `linstor sp l` reports the free
capacity cached at the satellite's last report rather than asking the
satellite, and that report can land while `zfs destroy` is still
freeing blocks in the background. A cleanup that had removed every
volume it created could then wait out its whole budget against a
figure that never moved. That residue has reached about 0.9 GiB, with
no known upper bound, against 1 GiB for the smallest volume a suite is
known to have leaked, so a byte tolerance has no safe value between
the two.

Record the datasets under each node's pool before the suite, read with
`zfs list` on that node's satellite, and wait until no dataset outside
that set remains. A leaked volume is then reported by name and node,
and a destroy still freeing space in the background does not count.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-02 02:17:00 +03:00
Mattia Eleuteri d8d9608e22 fix(backups): refuse a foreign -orig PVC and record the kept disk's reclaim policy
renamePVC returned early whenever a PVC named <dv>-orig-<hash> already
existed, without checking it came from this rename. The hash covers
only the RestoreJob name, so a RestoreJob created under the name of a
deleted one finds the -orig PVC its predecessor left behind. The rename
was skipped, the DataVolume deleted anyway, and the restore ran as if
keepOriginalPVC were false while reporting success: the live disk was
not kept, and the -orig name held an older one. Any other rename
failure ended the same way, since its error only became an event.

A -orig PVC now records the UID of the RestoreJob that created it. A
name held by any other PVC fails the restore before the VM is halted,
an interrupted rename of our own resumes instead of stopping halfway,
and a rename failure fails the restore before the DataVolume goes.

The PV is switched to Retain so it outlives the original PVC, and the
policy it had was discarded. Nothing in the controller deletes the
-orig PVC or restores that policy, so a PV that was Delete stayed
Retain and its volume leaked once an operator removed the claim. The
policy is now recorded on the PV and on the -orig PVC, and the restore
scenario documents how to set it back.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 22:00:19 +02:00
Mattia Eleuteri f514f8722b fix(backups, postgres): refuse a restore into another PostgreSQL major
A CNPG restore is physical: the recovered data files only open on the
major version that wrote them. The driver purged the target Cluster and
its PVCs without comparing the two, so a PostgreSQL 18 backup restored
into a v16 application, or in place into an application upgraded since
the backup, emptied the target. The recovery pods then looped on
"database files are incompatible with server" until the restore
deadline, and the RestoreJob failed with nothing left to fall back on.

The restore now fails before the purge, naming both versions, when they
differ. The target side is spec.version, since that is what the chart
renders the recovered Cluster from once the live one is gone. The
backup side is CNPG's status.majorVersion, copied into the artifact's
driverMetadata at backup time because retention deletes the
cnpg.io/Backup while the artifact stays restorable; artifacts taken
before that fall back to the cnpg.io/Backup. When either side is
unknown the restore goes ahead as before, so older backups keep
restoring.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 21:58:09 +02:00
Aleksei Sviridkin 30689f8109 fix(ci): map the recurring unmapped scopes and add area/dx
Over the full PR history a set of scopes recurring three or more
times still fell through to area/uncategorized. Most of them have an
obvious home and were only missing a row. Subsystem names such as
`networking` or `storage` are now accepted as the scope of their own
area, since contributors reach for those as often as for a package
name.

`dx`, `hack` and `cozypkg` describe developer tooling, which no
existing area covers: area/build is image builds, area/testing is
tests. They get a new area/dx rather than being filed under either.

The contract test also pins that every map target exists in
labels.yml, since the label sync workflow creates only what that file
defines, and that no key is written twice, since a repeated key
silently overrides the earlier one.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:37:22 +03:00
Aleksei Sviridkin fccb7fee06 test(tests): fail absence-only cases when the template renders nothing
Each of these templates is wrapped in a top-level condition, and the
case asserts only that a field is absent. If a values change turns the
condition off, the render is empty and the absence holds trivially, so
the case passes without looking at anything. A document selector makes
an empty render fail with "document not found".

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:27:17 +03:00
Aleksei Sviridkin ef5ef06806 test(tests): select documents by identity in absence-only test cases
These cases picked a document by position and then asserted only that
something is absent from it. When a template gains or reorders a
document, the index lands on a different object that lacks the field
just as well, and the case stays green while the property it names is
no longer checked. Selecting by kind or metadata.name keeps each case on
the object it names, and fails with "document not found" when that
object stops rendering.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:27:17 +03:00
Aleksei Sviridkin 77f6cd1af8 test(redis): assert no external SAN when no namespace map is injected
The null-namespace case only checked that the in-cluster name is
present, so a chart that emitted a bare "my-redis." SAN, or fell back
to the suite-level host, would still pass. It now carries the same
absence check as the empty-host case, plus the suite host itself.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:23:25 +03:00
Aleksei Sviridkin a8c3eb5260 fix(ci): strip every leading backport prefix from a PR title
A backport of a backport is titled with two `[Backport ...]` prefixes.
Only the first was stripped, so the second was parsed as a bracket
scope: the real scope and the kind were lost, and the area/release
added by the strip suppressed the area/uncategorized fallback, so the
PR looked labelled correctly.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:21:44 +03:00
Aleksei Sviridkin cd6476ccb7 fix(ci): look up the labeler's type and scope maps by own key only
Both maps are plain object literals, so a title typed or scoped
`constructor`, `toString` or `__proto__` resolved to an
Object.prototype member. That value is truthy, landed in the label
set and killed the step on `l.startsWith`, leaving the PR with no
labels at all.

The new contract test runs the workflow's own script under node with
stubbed context, so it exercises the code as shipped rather than a
copy of it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:21:39 +03:00
Aleksei Sviridkin 7c5d9d582b fix(installer): skip the packages artifact push and stamp with PUSH=0
image-packages ran `flux push artifact` regardless of PUSH, so a local
PUSH=0 build still published cozystack-packages and then stamped its
digest into values.yaml. The stamped platformSourceRef is the digest
the operator hands Flux to pull from the registry, so a build that
pushes nothing has no digest worth writing: it now skips both steps
and keeps the committed ref. With PUSH at its default of 1 the recipe
is unchanged.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:15:50 +03:00
Aleksei Sviridkin 4583ee595f fix(kubernetes-nodes): refuse a resources.cpu that cannot be a core count
On the kubevirt substrate a worker pool's resources.cpu, although
schema-typed as a Kubernetes quantity, is written verbatim into
domain.cpu.cores, which KubeVirt declares as uint32 and the
KubevirtMachineTemplate CRD as an integer. A value such as 500m or
"1.5" therefore rendered a template the API server refuses, and
because the same raw string was truthy it also counted as explicit
sizing and dropped the pool's instancetype matcher first. A whole
count past the uint32 ceiling rendered too, although KubeVirt cannot
hold it.

The render now decodes the value the way the emitted cores field is
decoded and stops unless that is a whole number between 1 and the
uint32 maximum, before the value reaches the VM template or the
precedence decision. Only inputs that could not produce a working
pool are refused. Every value that rendered one still renders the
same bytes, so no pool rolls: a bare numeric zero stays falsy and
means unset, and a leading-zero spelling is still emitted as written.

The proxmox substrate is left out of the check, because it converts
the quantity to millicores and rounds up to whole cores, so 500m is a
valid size there. The field description now says what the value
means on each substrate.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 22:10:40 +03:00
Aleksei Sviridkin 5fde704a30 fix(ci): skip directories named like Chainsaw step files
Chainsaw builds step files only from directory entries that are not
directories, so a directory named like 01-x.yaml makes nothing a
suite. Both selectors listed it as one, and select-install demanded a
mapping for it. A symlink still counts, as it does for Chainsaw. A
directory named chainsaw-test.yaml keeps its suite listed: Chainsaw
reads that name whatever its type and fails loudly on a directory, and
an unlisted suite would hide that failure.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 21:07:05 +03:00
Aleksei Sviridkin c309d08383 docs(ci): say what Chainsaw discovers under hack/e2e-chainsaw
The test-chainsaw comment said Chainsaw discovers every
chainsaw-test.yaml. It also loads chainsaw-test.yml, builds a test from
numbered step files, and walks subdirectories.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 20:35:41 +03:00
Aleksei Sviridkin e82e5baf65 fix(ci): discover Chainsaw tests nested below a suite directory
Chainsaw walks every subdirectory of the path it is handed, and the
runner hands it each top-level directory under hack/e2e-chainsaw/, so
a test in <dir>/<sub>/ runs as part of <dir>. Both selectors read only
one level down, so a directory whose tests all sit deeper was never
listed: no selection named it, the full-suite list left it out too,
and only a run with no selection at all executed it.

Both scripts now take the top-level directory of a test file found in
that directory or anywhere below it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 20:35:41 +03:00
Aleksei Sviridkin c1b99df98d fix(ci): escalate a file placed directly under examples/backups
A file beside the per-app backup directories rather than inside one
belongs to no single walkthrough, but it matched neither backup rule
and fell to examples/ in the inert list, so an edit to it selected
nothing and both lanes skipped Chainsaw. It is now treated as shared by
all of them and escalates the same way examples/backups/_lib/ does.
README.md there stays inert through the *.md rule.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 20:03:05 +03:00
Aleksei Sviridkin 6913a04622 fix(ci): discover Chainsaw suites made only of numbered step files
With no chainsaw-test.* in a directory, Chainsaw v0.2.15 builds a Test
from files named like 01-install.yaml. Such a directory runs whenever
the whole tree is handed to chainsaw, but neither selector listed it,
so no escalation carried it and no edit to it could select it.

Both scripts now derive the suite list from the file names Chainsaw
itself accepts.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 20:02:48 +03:00
Aleksei Sviridkin 5eee22fe28 ci(workflows): cache Go modules and fall back to direct in Go jobs
The unit and controller test job has no module cache and runs on an
ephemeral runner, so every run fetches every module from
proxy.golang.org, and one transient stream reset fails the job.

It now restores the module cache keyed on every go.sum in the tree and
prefetches the modules of every tracked go.mod through a bounded retry,
hack/go-mod-download.sh. The cache is written from main by the build
cache warmer under the same key, because a cache saved by a PR run is
visible to that PR only, while one saved on the default branch is
readable from every PR.

Every job that runs Go on the runner sets
GOPROXY=https://proxy.golang.org|direct so any proxy error falls back
to the origin. The release jobs get only that, not a cache.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Aleksei Sviridkin 548d33f01f build(images): fall back to direct and retry Go module downloads
A transient proxy.golang.org HTTP/2 stream reset fails an image build
outright: Go's default GOPROXY falls back to direct only on a 404 or
410. Image builds now get GOPROXY=https://proxy.golang.org|direct as a
build arg from hack/common-envs.mk, declared by every Go builder stage,
so any proxy error falls back to fetching from the origin. The value
comes from IMAGE_GOPROXY rather than the host's GOPROXY, which may
name a mirror that buildkit cannot reach. The alpine golang images
carry no git, so in their stages a module hosted on GitHub cannot be
fetched directly and the retry is the cover.

Explicit `go mod download` steps retry three times with a growing
pause. The loop is inline, in one form the contract test pins: the
images build from different contexts, so no shared script can be
copied in, and a named build context in the shared buildx arguments
makes buildx refuse every Dockerfile pinned to a frontend older than
dockerfile:1.4.

The two alpine stages that left the download to `go build` get the
same download step, because without git neither the fallback nor a
retry covered them.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 19:37:00 +03:00
Mattia Eleuteri f714ada567 fix(postgres): refuse a physical restore across image flavors
A recovery replays the source's data directory, so it carries the same
risk as switching the image of a live cluster: the postgresql and
postgis images are built on different Debian releases, and the glibc
change invalidates collatable indexes, while a postgis-to-postgresql
move also loses the libraries behind every PostGIS object. The chart
only guarded the live cluster, so a fresh release bootstrapped from a
backup of the other flavor went through, and the guard's own advice was
to restore a backup, which is that very move.

The backup-controller now records the source flavor in the snapshot it
stores on each Backup and fails a RestoreJob whose target runs another
flavor, before the purge deletes the target's Cluster and PVCs. A
snapshot without a flavor predates flavors and so comes from a
postgresql cluster. For the chart-managed bootstrap, the chart compares
against the recovery source's Cluster while it still exists. Both paths,
and the live-cluster guard, point a flavor change to a logical dump.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 18:31:18 +02:00
Mattia Eleuteri e4810d3268 fix(postgres): require a postgis image for every postgres major
The updater used to warn and leave a major out of postgis-versions.yaml
when the registry had no postgis build for its minor yet, while
versions.yaml still advertised it. `flavor: postgis` on that major then
failed to render. Postgis images trail the PostgreSQL ones, so this is a
routine state rather than an edge case: stop before writing either map
and let the next run pick up both once the image is published.

The version check now also compares the majors of the two maps, so a
map edited by hand cannot drop one silently. It is written in POSIX sh:
cozytest.sh sources it into /bin/sh, which is dash on the CI runners and
rejected the ${var//} expansions the first version used.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 18:30:37 +02:00
Mattia EleuteriandMatthieu dc2971264d feat(postgres): add a postgis image flavor
Tenants migrating spatial data need PostGIS, but the chart always runs
the CloudNativePG PostgreSQL image, which does not ship it, and exposes
no way to pick another one. CloudNativePG publishes a PostGIS operand
image; a closed `flavor` enum selects it without handing tenants a free
image field.

The flavor pins the same PostgreSQL minor as the default one, on the
standard-trixie variant: backups go through the barman-cloud plugin,
so the deprecated system variant is not needed. update-versions.sh now
derives the postgis map from the minors it picks, and a bats check
fails if the two maps drift apart.

The default images are built on bullseye and the postgis ones on
trixie. Switching a live cluster between them would swap glibc under
an initialised data directory and silently break every collatable
index, and CNPG would roll it out as a plain image update. The chart
therefore refuses a flavor change on an existing Cluster.

Co-authored-by: Matthieu <matthieu@hidora.com>
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-01 18:30:37 +02:00
Aleksei Sviridkin bc400dff15 test(e2e): hold the wait-budget guard to a start margin across four suites (#4468)
## What this PR does

`hack/mariadb-first-boot-waits.bats` let two regressions of the wait
budgets it guards pass green. This closes both. The change is test and
doc only: no wait, op or chart value moves.

A wait now has to clear its floor by a named `MBW_START_MARGIN` of 60s,
not by one second. The wait's clock starts before the install's by the
Application reconcile, the chart artifact fetch and the helm-controller
queue. Nothing in the tree bounds those three, so the value is a policy
choice and the header says so. It equals the headroom every current wait
already carries (660 against a 600s floor), which is why no value has to
change. A fixture pins 660 against 600 as passing and 659 and 601 as
refused. The sizing guide in `docs/agents/e2e-testing.md` asked only for
"strictly above" the floor, which the guard now rejects, so it now asks
for a start margin and points at the constant instead of restating the
number.

The mongodb, rabbitmq and clickhouse walkthrough waits and their
Chainsaw ops are now held to the same contract as mariadb, through one
table with a row per suite. Each suite gets its own floor test and its
own op test, so a regression turns only that suite's tests red. The
floor comes from the api-server's install-timeout default, and each
suite's ApplicationDefinition is checked for an override. The
startup-probe half is computed only for mariadb, since the other three
charts set no `startupProbe`. Each op must keep the remainder it had
before its waits were raised, and that raise preserved it: 900s for
mariadb and rabbitmq, 1200s for mongodb, 2700s for clickhouse. The
remainder is taken over every HelmRelease wait the op reaches, the
bucket wait included, so raising a wait together with its op keeps the
remainder instead of turning the raise into slack. The mongodb target
wait sits behind the `SKIP_RESTORE=1` its op passes, so it is not
reached and is left out; the test turns red once the op stops passing
it.

The Chainsaw HR-Ready asserts of mongodb, rabbitmq and clickhouse (6m)
stay out of scope. They sit under the floor by per-suite choice, and the
header says so.

Verification:

- The file goes from 22 to 28 tests. Four mariadb-only tests are
replaced by the eight per-suite ones: the example floor, the both-waits
op slack, the check that both applications are waited on and the
ApplicationDefinition override. Two fixture tests drive the per-suite
floor and op checks through directory, suite-file and RD overrides. They
cover each way of falling short that the live tree cannot produce, since
every live wait sits at exactly 660 and every remainder at exactly its
constant. That includes an RD that overrides or disables the install
wait, and a startup budget large enough to bind over the install
timeout.
- `make bats-unit-mariadb-first-boot-waits` passes, and the same file
passes under dash through `hack/cozytest.sh`, as CI runs it.
- Mutations, each confirmed applied and each run test by test, because
the runner stops at the first failure:
- Any suite's database wait lowered to 601: only that suite's floor test
fails. That includes the mongodb target wait the op never reaches.
- Each op lowered back to its old value (30m, 30m, 30m, 60m): only that
suite's op tests fail. For mariadb that is also the existing
prefix-remainder test.
- `SKIP_RESTORE=1` dropped from the mongodb op: only the mongodb op test
fails.
- Any suite's bucket wait raised to 660, or the rabbitmq one indented
out of the flow: only that suite's op tests fail. Raising a bucket wait
to 660 together with its op by 6m keeps all 28 tests green.
- A third clickhouse wait added in a new file under
`examples/backups/clickhouse/`: the clickhouse floor and op tests fail,
because the waits are read from every script in the directory rather
than from a listed subset.
- Any suite's target wait renamed to wait on the source again, in
mariadb and rabbitmq: only that suite's floor test fails. Dropping the
distinct-release check, or its `sort -u`, fails only the floor fixture
test.
- Inside the helpers themselves, one mutation per branch of every new or
changed helper, 37 in all: each turns its own fixture test red, or the
live suite checks where the branch selects which waits are read at all.
Two branches whose deletion changed no output were removed rather than
pinned.
- `MBW_START_MARGIN` set to 0: only the two fixture tests that encode
the margin fail. Deleting the constant fails every test that reads it,
as an unbound variable.
- `hack/select-e2e.sh` selects no suite for this diff.

Left for later:

- The file name. `hack/mariadb-first-boot-waits.bats` now guards four
suites. I kept the name because renaming it changes its make target and
every reference to it, which belongs in a change of its own.
- The SKIP_RESTORE stop works within one file. A suite whose waits span
several scripts, as clickhouse's do, would still count the waits in
later files if its op ever passed SKIP_RESTORE=1. That errs toward a
false red, not a silent pass, and no such op exists today.
- Measuring the margin. The delay from Application creation to the start
of the install is measurable from CI reports and would turn the 60s
policy value into a bound. That delay is not bounded anywhere today.

Closes #4418

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): the wait-budget guard now requires a 60s margin over each floor and covers the mongodb, rabbitmq and clickhouse backup walkthroughs alongside mariadb.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified that HR-Ready waits should exceed both the app’s first-boot
budget and the release install timeout, with a start margin for
controller processing.
* **Tests**
* Expanded wait-budget and timeout validation across configured suites,
including checks for missing or invalid settings, operation slack,
restore-skip behavior, and bucket waits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 18:42:59 +03:00
Alexey Artamonov 68ee97300b docs(monitoring): say a central tenant's collector still needs enabling
The tracingStorages description said a tenant with shared-central
tracing gets the collector without a local entry, which reads as if
tracingCentral alone deploys it. tracingCollector.enabled still
decides whether it renders.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-01 18:02:22 +03:00
Alexey Artamonov 69a16bd2da docs(tenant): shorten the tracingCentral description
The description ships in the godoc, the schema, the README, the
ApplicationDefinition and the dashboard's field help, and at about
2,800 characters it buried what an operator needs to decide on the
field. It now says what the field does, what tenant-root has to host
first, and who is inside the shared store's trust boundary; the
lifecycle details move to the user guide.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-01 18:02:22 +03:00
Aleksei Sviridkin a0af2fd2f8 test(e2e): hold the postgres and bucket walkthrough waits to the floor
The postgres backup walkthrough's HelmRelease waits and every table
walkthrough's bucket wait now clear the install-timeout floor, and the
Chainsaw ops grew to contain them, but nothing checked it: a wait
dropped back under the floor, or raised without its op, would pass.
The op check cannot catch the bucket case on its own, since a lower
wait only widens the slack.

Add postgres to the table the other walkthroughs are held by, with the
remainder its op kept through the raise, and hold each table suite's
bucket wait to the bucket release's floor, read from the api-server
default with bucket-rd checked for an override. This holds the budgets
only; what a CNPG first boot costs on this platform is still
unmeasured.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:22 +03:00
Aleksei Sviridkin 4386014cf4 test(e2e): restate the wait guard's figures for the raised bucket waits
The mariadb op and its bucket wait both grew by six minutes, so the
worked arithmetic in the remainder test and the note that the bucket
wait sits under its floor no longer describe the tree. The header also
said the other suites' HR-Ready asserts sit under the floor by choice,
which stopped being true once they were raised. The checks and their
constants are unchanged: every remainder still sits at exactly its
constant.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:22 +03:00
Aleksei Sviridkin 67e42bbbc0 docs(agents): stop describing the HR-Ready waits as under their floor
The sizing guidance said most of the tree sat under the install-timeout
floor, that mariadb was the only suite raised, that the walkthrough
bucket waits were under it, and that harbor's asserts were on and under
theirs. None of that holds once the waits clear their floors, and a
reader would take the old band as the norm to copy. Name kafka as the
floor no budget can clear, keep harbor as the example of two floors in
one suite, and note that a walkthrough wait passing no budget takes its
helper's default, which a grep for the number misses. The list of
definitions that disable the install wait also missed vm-instance-rd,
which is why its assert has no floor to clear, and the section never
said where a Package release's floor comes from.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:22 +03:00
Aleksei Sviridkin 04d469fc85 fix(tests): let the backup walkthrough waits outlive their installs
The backup round-trips drive examples/backups, whose bucket waits gave
up at 300s and whose postgres waits at 360s, under the 600s install
timeout cozystack-api gives each of those releases. The kafka example's
bucket wait passed no budget and took the helper's 300s default. A
failed install there reported only that the wait ran out.

Raise each to 660s, and grow the Chainsaw op around every walkthrough
by what its waits gained, so a failure at any later wait keeps the
headroom it had and is still reported by the script rather than cut
off by the op. The kafka and kafka-metadata Kafka waits stay: their
7h install timeout cannot be cleared inside the E2E job cap.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:22 +03:00
Aleksei Sviridkin 49cefe0dff fix(tests): size HR-Ready asserts on nested and Package releases
A nested <name>-system release takes its install timeout from the
template that renders it, not from cozystack-api, and a Package release
takes it from cozystack-operator. The external-dns, openbao and qdrant
templates set 10m and harbor's sets 15m; the operator gives 10m. Their
asserts sat at 5m or 10m, so a failed install reported as a bare
timeout. Raise each above its own floor: 11m, and 16m for
harbor-test-system. harbor-test was on its 600s floor exactly, which
expires first because its clock starts before the install's; it goes
to 11m with the generated parents here.

kafka keeps its 6m: kafka-rd sets a 7h install timeout that no budget
inside the E2E job cap can clear, and the assert now says so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:21 +03:00
Aleksei Sviridkin 9e6cb7d0cc fix(tests): let HR-Ready asserts outlive the install they wait on
cozystack-api gives every release it generates a 600s install timeout,
and these asserts gave up at 5-6m. A failed install was therefore
reported as a bare assert timeout while the release was still
installing and had not yet recorded why it failed.

Raise them to 11m, above that floor, so the release's own reason is on
record before Chainsaw gives up. The cost is up to six more minutes on
a red run of each suite; a green run is unchanged. vm-instance-test
stays: vm-instance-rd disables the install wait, so its release goes
Ready as soon as helm has applied.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:21 +03:00
Aleksei Sviridkin 94b328c6eb test(e2e): hold every raised backup wait to a start margin
The wait-budget guard accepted a wait one second over the install
timeout, although the wait's clock starts before the install's by the
Application reconcile, the chart fetch and the helm-controller queue. A
wait lowered from 660 to 601 passed every test and brought back the bare
assert timeout the budgets exist to avoid. A wait now has to clear its
floor by a named 60s margin. Nothing in the tree bounds those three
delays, so the value is policy: the headroom every current wait already
carries. The sizing guide in docs/agents/e2e-testing.md asked only for
"strictly above", which the guard now rejects; it points at the margin
instead of restating a number.

The mongodb, rabbitmq and clickhouse walkthrough waits were raised for
the same reason as mariadb's, and their Chainsaw ops grown with them,
but nothing held either. Each suite is now a row in one table and gets
its own floor and op tests, so a regression reds only its own suite.
Floors come from the api-server install-timeout default and, for
mariadb, the chart's startup probe; each suite's RD is checked for an
override instead of being assumed not to carry one. Each op keeps the
remainder it had before the raise over every HelmRelease wait it
reaches, the bucket one included, so raising a wait and its op together
keeps the remainder instead of widening the slack. The mongodb target
behind SKIP_RESTORE=1 is not reached and is left out until the op stops
passing it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 17:40:20 +03:00
Aleksei Sviridkin 889bb79236 feat(harbor): restrict the portal and API to listed source ranges (#4300)
## What this PR does

Adds `managementWhitelist` to the Harbor app, a list of IP addresses or
CIDR ranges allowed to reach the web portal and the Harbor API, and I
made it a typed list on purpose. The obvious alternative, a free-form
`ingress.annotations` map, gives tenants a way into the shared ingress
controller. The platform ingress-nginx runs with
`allowSnippetAnnotations: true` (the kubernetes app needs a
`configuration-snippet`), and v1.11.5 defaults `annotations-risk-level`
to `Critical`, so it accepts any annotation, `server-snippet` included.
Tenants can't create Ingress objects themselves, but they can write any
field of an app spec. An annotation map on the Harbor spec would let any
tenant put arbitrary nginx config into a controller that serves every
tenant. A typed list is also what the repo already does for this:
`filer.whitelist` in `extra/seaweedfs`, `whitelist` in `extra/ingress`
and `tcp-balancer`.

The use case is a registry that serves customers: pulls and pushes come
from anywhere, the admin UI and API should answer only to an office or
VPN range. Right now the portal, `/api/v2.0` and `/v2/` sit on one
Ingress and one public host, and a tenant can't restrict any of them.

With a non-empty list the existing Ingress gets
`nginx.ingress.kubernetes.io/whitelist-source-range`, and `/v2` and
`/service/token` move to a second Ingress, `<release>-registry-ingress`,
on the same host. ingress-nginx applies that annotation only to the
locations of the Ingress that carries it, so those two stay open. They
are what a registry client needs: Harbor answers an anonymous `/v2/`
request with a Bearer challenge whose realm is `/service/token` on the
same host. Everything else under `/service/` is
`/service/notifications/*`, internal job callbacks that the bundled
nginx already answers with 404, so it stays on the restricted side. The
second Ingress uses the same TLS secret and has no cert-manager
annotations, otherwise ingress-shim gets two owners for one Certificate.
The HTTP-01 solver is its own Ingress and is not restricted. With the
default empty list the chart renders exactly what it renders today.

An HTTPRoute can't express a source restriction, so in Gateway API mode
a non-empty list fails the render. Dropping it silently would leave the
portal public while the spec says it is restricted.

Each entry has to be an IP address or CIDR range, and a bad one fails
the render. ingress-nginx reads an empty `whitelist-source-range` as no
restriction and turns an unparseable one into a 503 for the whole
location, and the values schema generator can't validate list items.

Three things users should know. Harbor-to-Harbor replication that pulls
from this registry goes through its API, so the other registry's address
has to be on the list. The check uses the client address that
ingress-nginx sees: with `externalTrafficPolicy: Local` or PROXY
protocol that is the real client, behind another proxy it is the proxy.
And for the portal and API the list replaces the tenant ingress
`whitelist` rather than narrowing it.

Disabling the portal is not here. The vendored Harbor chart (1.19.1)
always renders the portal Deployment and its internal nginx always
proxies `/` to it, so there is no small switch to wire.

The new Ingress is added to the dashboard Role and to the ingress
selector in the ApplicationDefinition.

### Screenshots

No UI code change. The field reaches the dashboard form through the
regenerated schema and `keysOrder`.

### Downstream repositories

The terraform provider models Harbor by hand, so the new field needs a
follow-up there: cozystack/terraform-provider-cozystack#46. The website
reference page is generated from `README.md` on release. Nothing else in
the trigger map is touched: no package added or renamed, no platform,
installer or `hack/` change.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/46
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(harbor): add `managementWhitelist`, a list of IP addresses or CIDR ranges allowed to reach the Harbor web portal and API. Only `/v2/` and `/service/token` stay reachable from any address. The default is an empty list, which keeps the current behaviour. Not supported with Gateway API publishing: a non-empty list fails the render there. Each entry must be an IP address or CIDR range; for the portal and API the list replaces the tenant ingress `whitelist`.
```
2026-10-01 17:39:56 +03:00
Myasnikov Daniil 7fe713580d test(e2e): drop readiness asserts that HelmRelease Ready already covers
The qdrant and external-dns apps render their workload in a nested
-system HelmRelease, and both suites already assert that HelmRelease
Ready, which helm-controller reports only once the workload is up.
Asserting readyReplicas on the same Deployment or StatefulSet again
adds nothing, so those asserts keep only the instance-label pin.

The vminstance PVC asserts follow a DataVolume Ready assert, and a
DataVolume is not Ready before its PVC is Bound.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-01 18:03:25 +05:00
Aleksei Sviridkin 3303cc59c7 fix(e2e): delete the kafka suite's broker and controller PVCs
The kafka chart sets deleteClaim: false on both node pools, so deleting
an app leaves its volumes behind on purpose. The suite deleted its apps
and never the volumes. The 35 GiB a run leaves in tenant-test, 20 GiB
of it on one node, kept that LINSTOR node short of space for the
kubernetes suites that run after it. The backup round-trip leaves its
two clusters' volumes in tenant-root the same way.

Each test now deletes the PVCs by strimzi.io/cluster once its apps are
gone, and waits for them to go. The kafka test does it from a step
cleanup, so a failed run reclaims them too.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 15:27:18 +03:00
Aleksei Sviridkin 64b3c9619f fix(e2e): hold replicated-lane LINSTOR cleanup to a pre-suite baseline
The kubernetes suites' cleanup in the replicated lane demanded 90 GiB
free on every LINSTOR node. Volumes that earlier suites and the
platform legitimately keep can hold a node under that figure even when
the suite reclaimed everything it used, so the cleanup failed suites
that had leaked nothing.

Both lanes now record the per-node baseline before the suite creates
anything and compare against it on cleanup. The 90 GiB figure is what
a replicated suite needs to place its workers and CDI scratch volumes,
so it stays as a precondition checked before the baseline, and a
shortfall there is reported as the sandbox's state rather than the
suite's.

A run that fails before recording its baseline used to leave the
previous suite's file in place for its cleanup to compare against; the
file is now removed when the suite starts.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 15:27:18 +03:00
Aleksei Sviridkin 54159736bc feat(talos)!: bump Talos to v1.14.2 (#4598)
## What this PR does

Bumps the Talos image Cozystack builds from v1.13.6 to v1.14.2 to get a
newer DRBD. v1.13.6 ships DRBD 9.3.2, v1.13.7 and later and
v1.14.0/v1.14.1 ship 9.3.3, and v1.14.2 is the first release with 9.3.4.
ZFS moves from 2.4.3 to 2.4.4.

On 9.3.2 I've seen a peer reboot leave the DRBD sender thread spinning
on a failed send (EPIPE). What followed was RCU stalls, processes stuck
in D-state, a hanging `drbdsetup disconnect` and two-phase-commit
timeouts. The [9.3.4
ChangeLog](https://github.com/LINBIT/drbd/blob/drbd-9.3.4/ChangeLog) has
fixes for a sender thread pinning a CPU while its connection is down,
for two-phase-commit issues and a hanging `drbdadm disconnect`, and for
two nodes ending UpToDate with different data after a reconnect. None of
them are in 9.3.3. I matched the trace to those entries by their
descriptions only, not against the commits.

The base installer image had to change. Talos v1.14 no longer publishes
`ghcr.io/siderolabs/installer`, so `gen-profiles.sh` now uses
`ghcr.io/siderolabs/installer-base`, same as upstream's own v1.14 imager
profiles. I built the amd64 installer from the new profile with
`imager:v1.14.2` locally and it builds without errors.

PR CI will not boot the new image. It builds the installer and matchbox
images, but the e2e runs on the upstream `talos:v1.13.5` container from
`hack/e2e-compose.yaml`. The first run that boots the new kernel, DRBD
and ZFS is the nightly after merge, which builds the nocloud disk from
these profiles and runs it in QEMU.

On Talos v1.14.2 the `drbd_transport_tcp` module is no longer loaded on
demand. After upgrading a node from v1.13.6, `drbd` was loaded and the
transport module was not, `drbdsetup new-peer` failed with "Failed to
create transport (drbd_transport_xxx module missing?)", and every DRBD
resource on the node stayed in Connecting. Listing the module in
`machine.kernel.modules` fixed it without a reboot. The cause is
[siderolabs/pkgs#1565](https://github.com/siderolabs/pkgs/pull/1565), in
Talos since v1.14.0: the kernel is built with an empty modprobe path, so
it no longer loads modules on `request_module()`, which is how DRBD asks
for its transport. Any module that used to load that way now has to be
listed explicitly (reported upstream as siderolabs/talos#14501). Outside
DRBD this includes `dm-thin-pool` and `dm-multipath`, which the v1.14.2
kernel builds as modules, so LVM-thin or multipath users on Talos need
them listed too. The e2e node config now loads `drbd_transport_tcp`
explicitly, and the talm preset (cozystack/talm#252) and the install
docs (cozystack/website#720) get the same line. The module ships in the
drbd extension on v1.13 too, so the extra line is safe before the
upgrade.

The order matters for host nodes. Either update the cozystack preset in
the talm project to a version with cozystack/talm#252 and re-render and
apply the node config, or add `drbd_transport_tcp` to
`machine.kernel.modules` by hand. Only then upgrade the node to v1.14.2.
A node upgraded first loses DRBD replication until the module is added.

The host network policy now also denies port 2383 to `world`. Talos
v1.14 moved etcd's `/metrics`, `/health` and gRPC-gateway JSON API from
2379 to a separate listener on 2383, and the policy only blocked 2379
and 2380. Without this, the etcd JSON API on v1.14 control-plane nodes
would be reachable from outside, protected only by client mTLS.

Two smaller changes come with the bump. The multus
`talos-cni-plugins-checked-against` marker moves to v1.14.2, because the
pkgs release behind Talos v1.14.2 still pins CNI plugins v1.9.1, same as
multus. The system memory limits doc now describes the v1.14 OOM
trigger, since v1.14.0 dropped its global memory PSI clause
([siderolabs/talos#13895](https://github.com/siderolabs/talos/pull/13895)).

Before upgrading hosts, operators should know a few things from the
[v1.14.0 release
notes](https://github.com/siderolabs/talos/releases/tag/v1.14.0) and the
Talos compatibility code:

- Talos v1.14 accepts a host upgrade from v1.12.0 or later. It supports
Kubernetes 1.32 through 1.37, while v1.13 still accepted 1.31, so a
cluster on Kubernetes 1.31 has to upgrade Kubernetes first.
- etcd's HTTP endpoints (`/metrics`, `/health`) moved from port 2379 to
2383. The Cozystack etcd scrape proxy reads a separate metrics listener
on `127.0.0.1:2381`, which this change does not move. Anything else that
scraped or health-checked etcd on 2379 has to switch to 2383, and a
firewall outside the cluster that blocked 2379 should block 2383 too.
- etcd and kube-apiserver now require TLS 1.3, so a client that can only
speak TLS 1.2 stops connecting.
- Workload isolation (`SecurityProfileConfig`) stays off on upgraded
clusters, but `talosctl gen config` turns it on for new ones. I haven't
tested Cozystack with it enabled.
- `talosctl apply-config --mode=reboot` is gone.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#720
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [x] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
cozystack/talm#252
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website PR refreshes the `next` version pins for v1.14.2 and adds
the module to the Talos install pages. The talm PR adds it to the
cozystack preset.

### Release note

```release-note
feat(talos): Bump Talos to v1.14.2, which ships DRBD 9.3.4 and ZFS 2.4.4. DRBD 9.3.4 fixes a sender thread spinning on a dead connection, two-phase-commit hangs and a data divergence after a reconnect. Before upgrading a node, add `drbd_transport_tcp` to its Talos `machine.kernel.modules` next to `drbd`: Talos v1.14.2 no longer loads it on demand, and without it DRBD cannot connect to any peer. We recommend skipping the v1.13 line and upgrading nodes straight to v1.14.2. Talos accepts a direct upgrade from v1.12.0 or later. Talos v1.14 moves etcd's HTTP metrics and health endpoints from port 2379 to 2383, and the Cozystack host network policy now blocks 2383 from outside the cluster.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Updates**
* Upgraded host node support to Talos v1.14.2 across machine images,
with refreshed firmware and storage extensions.
* Updated memory-pressure guidance for Talos v1.14, including
version-specific behavior and configuration recommendations.
  * Added protection for Talos v1.14’s etcd HTTP endpoint on port 2383.
* Updated cluster preparation to load the TCP transport module required
for DRBD.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:30 +03:00
Aleksei Sviridkin de570a3806 feat(platform): add slim installation variants (#4595)
## What this PR does

This adds three minimal platform variants for small installs, such as
arm64 labs, where isp-full is too heavy. They are `isp-slim` for Talos,
`isp-slim-generic` for k3s, kubeadm or RKE2, and `isp-hosted-slim` for
clusters where the host provides CNI and storage.

A slim variant installs only the base platform: the engine, API and
dashboard, tenants, ingress and Gateway API. The two non-hosted variants
also get LINSTOR. objectstorage-controller stays on too, because
cozystack-controller watches BucketClaim unconditionally and crash-loops
without that CRD. Everything else is opt-in through
`bundles.enabledPackages`, including every paas and naas application and
its operator, monitoring, backups, etcd, SeaweedFS, metrics-server, VPA,
Multus and MetalLB. The iaas bundle is refused, so there are no VMs and
no managed Kubernetes. With the stock preset a slim variant emits 20
Packages (15 for hosted), where isp-full emits 77.

On the two non-hosted slim variants, networking is Cilium alone. Without
VMs nothing needs Kube-OVN, and Cilium L2 announcements take the place
of MetalLB. The admin creates a `CiliumLoadBalancerIPPool` and a
`CiliumL2AnnouncementPolicy`, or uses `publishing.externalIPs`. Pod
ranges come from `node.spec.podCIDR`, which Talos and k3s allocate by
default. `networking.encryption` only drives Kube-OVN IPsec, so the slim
variants refuse it instead of silently ignoring it.

The tenant application needs its own slim variant. The default one waits
on the monitoring, etcd and SeaweedFS applications, so on slim
cozystack-basics and tenant-root would never become ready. tenant-rd and
cozystack-basics still reference the default variant's artifacts by
name. Those artifacts exist anyway, because the PackageSource builds
them for every variant.

The full variants don't change. I rendered isp-full, isp-full-generic
and isp-hosted against main and got byte-identical Packages. The only
new output there is the slim variant in the tenant-application
PackageSource.

Things to know before using it:

- An opt-in package does not pull in its dependencies. The presets and
the docs list the chains. A package enabled without its chain stays
`DependenciesNotReady`.
- Slim is for new installs. Switching a live isp-full cluster to
isp-slim moves the networking Package from `kubeovn-cilium` to `cilium`,
the operator removes the Kube-OVN release, and running pods lose
networking. The other Packages are kept by `helm.sh/resource-policy:
keep`, so the switch doesn't make the cluster smaller either.
- No e2e suite runs a slim variant yet. Helm unit tests cover the
presets, opt-in, OIDC, the networking values and both refusals, and a
dependency check over every emitted Package passes for all three
presets.

I installed isp-slim from a build of this branch on a fresh three-node
Talos 1.13 cluster (amd64). All Packages and HelmReleases went Ready,
with no Kube-OVN, MetalLB, monitoring, backup or KubeVirt pods. From
outside, the dashboard and the API answered by name with valid
certificates. A LoadBalancer Service got an address from a Cilium pool
and answered through L2 announcements. Pod-to-pod traffic across nodes,
cluster DNS and a replicated LINSTOR volume worked. Opting in
postgres-operator and postgres-application gave a Ready Postgres. That
run found the BucketClaim crash-loop above, which is fixed here. The
base platform used about 215m CPU and 3.2 GiB of memory, not counting
the Kubernetes control plane.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#718
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [x]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up: cozystack/ansible-cozystack#79
- [x] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
cozystack/ccp#23
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(platform): add the isp-slim, isp-slim-generic and isp-hosted-slim variants. They install only the base platform (engine, dashboard, tenants, ingress, gateway, and LINSTOR outside hosted) with Cilium-only networking and no MetalLB; applications, operators, monitoring, backups, etcd and SeaweedFS are opt-in through bundles.enabledPackages, and the iaas bundle is not available.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added the `isp-slim`, `isp-slim-generic`, and `isp-hosted-slim`
platform variants, with support for PaaS and NaaS bundles.
* Slim variants use Cilium networking and include a reduced set of
packages by default. Additional packages and their dependencies can be
enabled as needed.
* **Limitations**
* IaaS and the platform encryption toggle are not supported on slim
variants. Cilium-native encryption is not configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:22 +03:00
Aleksei Sviridkin 312409ecd2 test(e2e): fail the suite whose pods outlive its cleanup (#4492)
## What this PR does

A Chainsaw suite could pass while the workloads it created kept running
after its own cleanup, and nothing noticed until a later suite failed.
On one E2E run the brokers of four Kafka releases from suites that had
passed and deleted their applications (#4488), plus a TenantControlPlane
sitting on its finalizer with two apiserver pods (#4489), pushed pod
memory requests to the node limit. Harbor then failed on `Insufficient
memory` two suites later, under the wrong name.

This makes the suite that leaked fail, and name what it left behind.

- Chainsaw v0.2.15 has no hook that runs after a test's cleanup: its
`Configuration` carries `error.catch`, which runs only on failure. So
`make test-chainsaw` now runs `hack/e2e-chainsaw-run.sh`, which invokes
`chainsaw test` once per suite directory and checks the cluster between
invocations with `hack/e2e-chainsaw/_lib/leak-guard.sh`.
- Attribution: a pod is charged to a suite when all three hold. It is
not terminal. Its UID was not in the pod list taken before the suite.
The root of its ownerReferences chain was created at or after the suite
started. A platform pod replaced mid-suite by a rollout, an eviction or
a crash traces back to an object the install created, so it is not
charged. A pre-existing pod adopted by a controller recreated mid-suite
is excluded by the UID baseline. A leak is charged once, to its own
suite.
- The suite start is read off an apiserver, from the `creationTimestamp`
of a server-side dry-run create, rather than off the sandbox. With
several control-plane nodes the reading and a root's timestamp can come
from different apiservers, so the rule assumes their clocks agree more
closely than the gap between the reading and the suite's first object;
no skew margin is added. Source for that behaviour: the generic registry
`Store.create` stamps it through `rest.FillObjectMetaSystemFields`, then
`DryRunnableStorage.Create` copies it back without persisting
(`staging/src/k8s.io/apiserver/pkg/registry/generic/registry/store.go`
and `dryrun.go` at Kubernetes v1.33.0).
- Budget: charged pods get 300s to disappear, the default
`timeouts.cleanup` in `.chainsaw.yaml`; a unit test keeps the two equal.
A suite that overrides its own cleanup timeout (kafka sets 8m) is
covered the same way, because the check runs only after `chainsaw test`
has returned, which is after every cleanup in the suite has finished
under its own timeout; the 300s start from there. The leaks this targets
sat about 110 minutes (Kafka brokers) and 30+ minutes (the
TenantControlPlane) past their suites. Every green suite logs
`leak-guard: suite=<dir> clear after Ns`, so the number can be checked
against later runs.
- Past the budget the run prints `LEAKED-PODS suite=<dir> budget=300s
count=N`, then one line per pod with its phase, deletion state, root and
`app.kubernetes.io/instance`. It raises a `leaked-pods` error annotation
and adds a failed `leak-guard` test case to `chainsaw-report.xml`. The
run continues through the remaining suites and exits non-zero at the
end, as the single invocation did on a red test.
- A suite that already failed on its own is checked once without the
wait. Its leftovers are reported as a `::warning` with `LEAKED-PODS
suite=<dir> checked-once=chainsaw-failed`, since some may still be
tearing down.
- Each pod gets one `kubectl wait` inside the remaining budget, and
nothing is retried. A timeout or a spent budget gives the verdict from
the last good pod list, also when the re-list after it fails, so a leak
fails the suite. A wait that fails only after running for its whole
`--timeout` counts as a timeout whatever it printed: kubectl maps an
expired context to "timed out waiting for the condition" only inside the
watch, so a deadline that passes during its initial lookup surfaces as
"context deadline exceeded" (`wait.go` and `delete.go` in kubectl
v1.33.0). A spent budget is caught before calling kubectl and never
passed as `--timeout=0`, which means "check once".
- Any other error on a wait or a re-list, before the budget is spent and
before any timeout, stops the waiting and is reported as `LEAKED-PODS
suite=<dir> count=N inconclusive="<error>"`. That is a warning, not a
leak failure, so an apiserver blip cannot become a leak red.
- A failed first reading after a suite, including a pod list that cannot
be parsed, leaves the suite unchecked. It fails with a `leak-guard`
annotation and a report case saying the cluster could not be read, not
that it leaked. Errors that are not leaks carry the annotation title
`leak-guard`, not `leaked-pods`.
- Known leaks are listed in `hack/e2e-chainsaw/_lib/leak-allowlist.txt`
as `<suite-dir> <issue-ref>`: `kubernetes-latest` and
`kubernetes-previous` against #4489 (both suites create a
TenantControlPlane through `_lib/run-kubernetes.sh`). `kafka` and
`kafka-metadata` were listed against #4488 until #4490 fixed it; those
entries are gone. An allowlisted suite prints its leak as a `::warning`
and does not fail. An entry that names a directory that is not a suite
stops the run before the first suite, so a rename cannot keep an
exemption alive. The PR fixing each issue deletes its lines.

Suite selection is unchanged. The runner runs the named directories when
there are any, and otherwise every directory directly under
`hack/e2e-chainsaw/`, each handed to Chainsaw's own discovery; it
refuses a tree with a test at its top level, where that split would stop
being exact. I verified this with Chainsaw v0.2.15's own
`discovery.DiscoverTests`, built from source at that tag. For the full
tree, for `kafka redis harbor` and for `kubernetes-latest`, the union of
the per-suite discoveries equals what a single `chainsaw test [suites]`
discovers: 57, 7 and 2 tests, identical lists. 57 matches the test count
in an earlier full run's report. A unit test pins that the runner passes
chainsaw the suites and flags it was given, in order. The order changes
too: suites used to run in Chainsaw's own order and now run one
directory at a time in alphabetical order, which moves the kubernetes
suites after the kafka ones. Before #4490 fixed #4488 this starved them
(see the E2E criterion below). With the fix merged the order has no
effect, and a future leak that lands before a memory-heavy suite is
named at the suite that leaked. One behaviour does change: a selected
name that is not a directory used to fail the single invocation before
any test ran; now only that suite fails and the others run.

`chainsaw-report.xml` keeps its path and shape: one `<testsuites>` root
with the summed counts, one `<testsuite>` per suite directory. Its
consumers are the `Collect chainsaw report` / `Upload chainsaw report`
steps in `pull-requests.yaml`, `e2e-fork.yaml`, `nightly.yaml` and
`e2e-tag.yaml`, which `docker cp` the file and upload it as an artifact.
Nothing in the repository parses it. The merge is unit-tested on two
per-suite reports in Chainsaw's real shape, one of them failed, and the
result is checked to parse as XML.

Tests: `hack/chainsaw-leak-guard_test.bats` (31 tests, in `make
unit-tests`; named without the `e2e-` prefix, which the unit lane
excludes) covers the attribution rule, the budget, the allowlist, the
merge and the runner, all against stub `kubectl` and `chainsaw`. Every
rule was mutation-checked: removing the guard call, having the Makefile
bypass the runner, inverting the allowlist match, flipping or tightening
the root-timestamp comparison, removing the UID baseline, removing the
terminal-phase filter, skipping the budget wait, changing the budget,
lowering or raising the depth cap, accepting a stale allowlist entry,
reading an unreadable pod list as empty, dropping the leak case from the
report, skipping directories in the selection, treating any failed owner
read as a missing owner, dropping the separate report for a suite that
already failed, dropping the not-run reason from the report, reading any
wait error as a timeout, removing the inconclusive branch, letting a
failed re-list after a timeout turn a leak inconclusive, ignoring a
failed re-list, reporting an unreadable cluster as a leak, reading an
unparsable pod list as clean, reporting a kubectl warning instead of the
error, passing the report name without its extension, wording a failed
re-list as a failed wait, dropping an allowlist's last line when it has
no newline, dropping the check that `--set-string` carries a value, and
reading a wait that ran its whole timeout but printed a context error as
inconclusive. Each one turns its named test red, and restoring turns it
green.

E2E criterion, registered before the run: in the `Run E2E tests` log of
this PR, every suite that ran prints `leak-guard: suite=<dir> clear
after`, a `LEAKED-PODS suite=<dir>` line carrying `allowlisted=`, or one
carrying `checked-once=` (a suite that failed on its own) or
`inconclusive=` (waiting stopped on a failed read), and no line matches
`LEAKED-PODS suite=.* budget=`. With #4490 merged, `kafka` and
`kafka-metadata` are no longer allowlisted and must print `leak-guard:
suite=kafka clear after` and `leak-guard: suite=kafka-metadata clear
after`, which is the live check that #4490 fixed #4488. The first run of
this PR (before #4490, run 36223896793) is the positive control: it
printed `LEAKED-PODS suite=kafka count=15` and `suite=kafka-metadata
count=8`, allowlisted, and every other suite cleared in 0-37s. That run
also went red on both kubernetes suites, because those 23 leaked brokers
left no memory for the second tenant worker VM (`0/3 nodes are
available: 3 Insufficient memory`), which the order change described
above made possible. The guard has not yet run against a real cluster,
so the first run of this PR should carry the `full-e2e` label; its
`clear after` lines are the first measured teardown times, and any
unexpected `LEAKED-PODS` line is either a real leak or a false positive
to understand before merge.

Follow-ups, known and out of scope here:

- The allowlist exempts a whole suite, so a new leak in an allowlisted
suite stays hidden until the issue's fix removes the line.
- Attribution is per suite directory, not per Chainsaw `Test`; the leak
lines carry the release instance, which usually names the test.
- Only pods are checked. A leaked PVC, LoadBalancer IP or bucket holds
no node memory and is not seen.
- The 300s budget rests on Chainsaw's cleanup timeout, not on a measured
pod-gone latency; the `clear after` lines are where that measurement
will come from.
- A platform object recreated mid-suite (for example a HelmRelease
uninstall/reinstall remediation, or a force-replace on an immutable
field) has a root newer than the suite start, and its pods would be
charged to whichever suite was running as a leak; the first CI runs are
where to look for this.
- The per-pod wait recomputes the remaining budget before each pod, and
no unit test pins that.
- `kubernetes-previous` is allowlisted for #4489 although the stuck
finalizer was seen only in `kubernetes-latest`; both suites create a
TenantControlPlane the same way, and the exemption hides any new leak
there until #4489 is fixed.
- Each clean suite now waits for its own pods to finish terminating
before the next starts, where the tail used to overlap the next suite.
The first full run's `clear after` lines say what that costs against the
E2E job's 215-minute cap.
- After a timeout followed by a failed re-list, the leak lines come from
the list taken before the wait, so a pod whose own wait had already
succeeded is still named and the count can be too high; the verdict is
unaffected.
- Likewise, when every wait succeeded and the re-list then fails, the
inconclusive report still lists the pods whose deletion was just
confirmed. In the boundary case where the budget runs out during the
last successful wait and the re-list then fails, the result is
inconclusive rather than a verdict from the last good list, because the
guard only treats the budget as spent when it finds it exhausted at a
check.
- A mistyped `COZY_LEAK_ALLOWLIST` path passes allowlist validation; it
fails closed, since no suite is then exempt.
- An allowlisted suite is not waited on, so `kubernetes-latest` and
`kubernetes-previous` will print `LEAKED-PODS ... allowlisted=` on most
runs even for pods still in normal teardown; waiting out the budget
first and applying the allowlist only to what remains would stop that
noise.
- A slow teardown in each of several suites can add up to 300s apiece,
against the E2E job's 215-minute cap. On the first full run of this PR
(run 36223896793) the `clear after` values of the suites that were
waited on summed to 210s; the last full run before this change took 158
of the 215 minutes.
- A pod whose owner's resource type is no longer served gets "the server
doesn't have a resource type" rather than NotFound, so its suite is
reported unchecked instead of leaked; it still fails.
- `kubectl wait --for=delete` waits for deletion, while the guard skips
pods in `Succeeded` or `Failed`; a pod that finishes but is never
deleted (a Job under a leaked root) spends the rest of the budget before
the suite shows `clear`. The verdict is right; only job time is lost.
- The unit tests do not pin the one-line empty report Chainsaw writes
for a directory with no tests (`_lib`), nor the branch that catches a
spent budget before calling `kubectl wait`; both are verified against
the Chainsaw and kubectl sources rather than by a test.
- The runner's per-suite report directory from `mktemp -d` is not
removed; the sandbox is discarded after the job.
- Each non-suite directory under `hack/e2e-chainsaw/` (today `_lib`)
costs one empty Chainsaw invocation, because the runner leaves the
decision of what is a test to Chainsaw.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds files under `hack/`, changes the `test-chainsaw` target,
and rewords one comment and the failure line in the four E2E workflows.
The trigger map's `hack/` entry is for cozystack/ccp, whose skills
anchor on `hack/package.mk`, `hack/common-envs.mk` and `make generate`,
none of which change here. A GitHub code search for `chainsaw` across
the cozystack organisation finds hits only in this repository.

### Release note

```release-note
test(e2e): the E2E app suite now fails the suite whose pods outlive its own cleanup by more than 300s, naming each leftover pod and its owner; suites with a filed leak are listed in hack/e2e-chainsaw/_lib/leak-allowlist.txt and only warn.
```
2026-10-01 14:03:13 +03:00
Aleksei Sviridkin 42b7239198 test(e2e): wait for the opensearch handoff before the first API call (#4462)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The opensearch suite sends its first OpenSearch API call, the index
create in `make-the-cluster-permanently-yellow`, as soon as the nodes
StatefulSet reports one ready replica. Two operator actions are released
at that moment and neither has finished: the bootstrap pod handing over
the cluster-manager role, and a rolling restart of the only data node.
#4412 is the first of them hitting the create with a 503. This PR makes
the step wait for both before its first call: an `error:` assert that
`opensearch-test-bootstrap-0` no longer exists, then an assert that the
nodes StatefulSet has settled on its update revision, in the same form
the suite already uses after the listener move.

Anchors below are opensearch-k8s-operator v2.8.0 with the in-tree
patches applied, paths under `opensearch-operator/`.

**Bootstrap handoff.**

- The operator starts retiring the bootstrap pod only once the cluster
is `Initialized` (`pkg/reconcilers/cluster.go:110-111`), and that flag
is set as soon as every cluster-manager pool has all its replicas ready
(`controllers/opensearchController.go:257`,
`pkg/builders/cluster.go:1146-1164`). A Ready data node therefore starts
the handoff and does not finish it.
- The voting exclusion is a cluster-manager action, so it goes through
the bootstrap pod and fails while that pod's security is not
initialized. The operator then keeps the pod and requeues after 10s
(`pkg/reconcilers/cluster.go:36`, `:186-190`).
- The pod is deleted only after the bootstrap node has left the last
committed voting configuration (`pkg/reconcilers/cluster.go:195-205`,
`opensearch-gateway/services/os_client.go:760-771`). Once the pod object
is gone, including its Terminating phase, the bootstrap process has
exited and holds no vote, and the only cluster-manager-eligible node
left is `opensearch-test-nodes-0`.
- The assert cannot pass vacuously. The cluster forms around the
bootstrap pod (`cluster.initial_master_nodes` names only that pod,
`pkg/builders/cluster.go:807`), and the pod is created unconditionally
while the cluster is not initialized
(`pkg/reconcilers/cluster.go:112-113`).

**Rolling restart on first boot.** The failing run's own operator log
and StatefulSet dump show it:

- `10:27:52.850` the reconcile in which `Initialized` became true.
- `10:27:53.251` "Failed to exclude the bootstrap node from the voting
configuration".
- `10:27:53.981` "Starting rolling restart of the StatefulSet
opensearch-test-nodes" (`pkg/reconcilers/rollingRestart.go:155`).
- `10:27:54.079` "Couldn't proceed with rolling restart … because
enabled shard allocation" (`:182`), the first step of the yellow path,
after which the next 10s requeue moves on to `PreparePodForDelete`
(`:195`) and deletes the pod.

The StatefulSet in the same dump is at generation 2 with
`currentRevision` `…744cc7b944` and `updateRevision` `…677c65d7fb`: the
pod template was rewritten after `opensearch-test-nodes-0` was created,
and the restart onto it waits only on `Initialized`
(`rollingRestart.go:79`, `:123`). With one replica that restart takes
down the node the step's requests go to, so a create that got past the
handoff could still land on a terminating or restarting node. Once the
StatefulSet reports `currentRevision == updateRevision` with its one
replica updated and ready, the node the step talks to is on its final
template and the operator has nothing pending.

The suite runs one topology: one replica in a single pool with the
cluster_manager, data and ingest roles, on a PVC. The bootstrap pod
comes back only through the emptyDir recovery path
(`pkg/reconcilers/cluster.go:452-478`), which a PVC-backed cluster never
takes, and a restart that was not pending leaves the second assert true
immediately.

I checked the rest of the suite for the same window. Apart from this
step, only the catch script (diagnostics, `|| true`) calls the
OpenSearch HTTP API. `verify-endpoint-serves-the-chart-chain` makes a
TLS handshake only, and it runs after the suite has already waited for
the listener roll to complete, on a single node that is its own
cluster-manager.

Budgets: 3m for the handoff, which took about two seconds in the failing
run; that leaves room for several 10s retire retries, a 30s requeue and
the default 30s termination grace period the bootstrap pod runs with. 6m
for the restart, shorter than the 12m the suite gives the listener roll
because nothing is pulled here: the health steps requeue every 10s and
the pod then restarts on an image the node already holds, where the
first boot, bootstrap included, took under two and a half minutes.

Residual risk, unproven: if the pod template were rewritten again after
the rollout assert passed, that assert would have passed on a template
that is not the final one. I found no mechanism for it. The StatefulSet
builder reads no cluster status (the only status read in
`pkg/builders/cluster.go` is the replica check in `AllMastersReady`), so
the transition to `Initialized` cannot rewrite the template by itself; a
rewrite would need a configuration or secret input to change after that
point. Which input changed on first boot in the failing run I could not
tell from the log, because the ControllerRevisions were not captured.

What I ran: `chainsaw lint test` with Chainsaw v0.2.15, the version the
e2e sandbox pins, passes on the changed suite. As a positive control,
lint rejects each new operation when I rename one of its keys.
`hack/select-e2e.sh` on this diff selects `opensearch` only.

What I did not run: the E2E suite itself. The race is timing-dependent
(one hit in six failed in-tree runs checked, per the issue), so a single
green run would not prove the fix either. The argument rests on the
operator code and the failing run's log cited above.

Closes #4412

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The only file changed is
`hack/e2e-chainsaw/opensearch/chainsaw-test.yaml`, which no downstream
repository consumes.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Cluster health checks now wait for bootstrap completion and verify
that all nodes are ready and up to date before proceeding.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:03:01 +03:00
Aleksei Sviridkin 4d7f9e485a docs(ci): say why the overlay's ancestry tie goes to main (#4469)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

This fixes the comment on the overlay's ancestry tie, which described
the wrong reason for the tie. No behaviour changes.

The comment said a line fresh off main is the same as main. That holds
only for the graph. `cozystack-packages:main` is built from main's tip,
so once main moves past the cut, a stack on such a line gets newer
images than the line. The comment now says so. It also says why the tie
still goes to main: a branch forked from main before the cut ties the
same way, and that case is far more common.

This follows up on a review note in #4460 and is stacked on it.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
docs(ci): explain why the base-branch overlay sends an ancestry tie to main.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified how artifact selection works for stacked pull requests on
release lines without line-specific commits, including why `main` is
preferred in a tie and how that may affect image freshness.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 14:02:54 +03:00
Aleksei Sviridkin a66907f535 fix(backups): share the walkthrough wait helpers and fix the S3 preflight cleanup (#4635)
## What this PR does

This fixes the S3 access preflight and a few backup walkthroughs, and
moves the walkthrough wait helpers into one shared file.

The access preflight now keeps its evidence and cleans up after itself.
When `mc alias set` failed, it dropped the port-forward log, and now it
prints it like the other exits do. The exit trap used to kill the
port-forward before `mc rm`, so the delete always hit a closed port and
the warning gave no reason. The delete now runs first, and the warning
carries mc's own error, or says the delete timed out when mc was killed
before writing one. A PUT that timed out (rc 124) may have landed, so it
counts as uploaded. A PUT the server refused does not, so access that
never went live gets no delete and no warning.

The wait helpers were copied into every walkthrough and had drifted.
`wait_for_field`, `wait_hr_ready` and `wait_deleted` had 12, 8 and 2
copies with 5, 3 and 2 different bodies. They now live in
`examples/backups/_lib/wait-helpers.sh`, and each `00-helpers.sh`
sources it. Call sites and wait budgets are unchanged. Where copies
differed, the shared body keeps every diagnostic any copy had. What
changes per walkthrough:

- `wait_for_field` in every walkthrough but redis now shows kubectl's
stderr, so a Forbidden or transport error is visible instead of reading
as an empty field. `--ignore-not-found` keeps a resource that does not
exist yet from printing an error on every poll. Redis already kept
stderr.
- `wait_for_field` in foundationdb, nats and vmi gains the optional 7th
terminal-value argument. None of their callers pass it, so nothing
changes at runtime.
- vmi loses its progress dots.
- `wait_hr_ready` in redis and kafka-metadata now prints every condition
and the release history on Stalled or timeout. It also says whether the
release never appeared, was deleted, or could not be looked up.
- The kafka `wait_deleted` timeout message reads "it is still present"
instead of "still present".

`hack/backup-examples-shared-helpers.bats` goes red if a walkthrough
defines one of these helpers again or stops providing them, and it pins
both sides of the `wait_for_field` read: kubectl's reason for a failed
read is shown, a missing resource prints nothing. The byte-identity pin
between copies in `hack/mariadb-first-boot-waits.bats` is gone, and its
errexit test runs through the mariadb walkthrough's helpers.

The kafka topic-data demo had the empty-read race from #4400.
`kafka_run` read topic state through a throwaway `kubectl run -i` Pod.
It now execs into a long-lived CLI Pod that keeps the restricted
security context, and `cleanup.sh` removes it by label. The Pod name
comes from its own `KAFKA_TOPIC_CLI_POD`, so setting the kafka-metadata
demo's `KAFKA_CLI_POD` does not move it. #4400 itself was already fixed
on main for nats, kafka-metadata and rabbitmq.

The nats and kafka-metadata demos could exec into a Pod they had just
refused. They read CLI output inside `$(...)`, where bash does not apply
errexit, so the ownership refusal was ignored. A failed read of the
Pod's phase was treated as no Pod, and the demo went on to delete and
replace it. Every setup step now returns on failure. Their "exec fails
with kubectl's own error" tests could never fail: the runner's trace
wrote the expected message into the same stderr the test grepped. The
stub now prints the message itself.

Each fix has a test that fails on main. For each fix I also reverted it
and checked that only its own test goes red.

One gap stays open outside this PR. `hack/select-e2e.sh` maps
`examples/backups/<app>/` paths by app name, so a later PR that changes
only `examples/backups/_lib/wait-helpers.sh` would select no suite. This
PR is fine because it touches every walkthrough. The rule belongs to the
select-e2e owners and is tracked in #4626.

While reviewing I also found that the kafka README puts steps that
create and exec into Pods under the tenant persona, and the tenant role
can do neither. That is #4625, left for its own change.

Left out on purpose, because they are outside what this diff changes:

- The kafka topic demo and the kafka-metadata demo each carry a CLI Pod
helper, and the kafka-metadata one runs without the restricted security
context.
- The walkthrough READMEs do not say that a directory now needs
`../_lib` next to it.
- The errexit test in `hack/mariadb-first-boot-waits.bats` loops over
one file. I kept it so the rebase against #4471 stays mechanical.

Closes #3757, closes #4243, closes #4244

### Screenshots

No UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup walkthroughs share one set of wait helpers that keep kubectl errors visible, the kafka topic-data demo no longer reads empty replies from a throwaway Pod, the nats and kafka-metadata demos no longer exec into a CLI Pod they refused, and the e2e S3 access preflight says why client setup or cleanup failed and deletes its test object reliably.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Kafka topic backup walkthroughs now reuse a named CLI Pod, with
restricted security settings and checks to prevent using a Pod owned by
another demo.
* Backup walkthroughs can wait for resources to reach a desired state,
Helm releases to become ready, or resources to be deleted.

* **Bug Fixes**
* Kafka and NATS walkthroughs now stop and report errors when CLI Pod
lookups, creation, deletion, or command execution fail.
* S3 preflight cleanup now attempts removal after a timed-out upload and
reports cleanup failures, including timeouts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 13:17:28 +03:00
Aleksei Sviridkin a9dc106a81 fix(e2e): bound post-install-prep calls and make its failures name their cause (#4632)
## What this PR does

The post-install-prep step of the e2e install could hang without
failing, and several of its failure paths gave up without saying why.
The script runs in the background of the install test, which waits on
it, so a kubectl call that never returns keeps the install open until
the job's own timeout. This PR fixes both problems in
`hack/e2e-post-install-prep.sh`.

- Every call that does work now has a wall-clock bound: the pool listing
and creates, both applies, the CDI patch, the reads in the LINSTOR node
loop, and the diagnostic reads on every failure path. Plain requests
also get a smaller `--request-timeout`, so kubectl can say why a request
failed before the outer timeout kills it. When a bound fires, the log
names the call.
- `kubectl wait --timeout` turned out to be no bound at all for the GET
it runs first, so each wait attempt got an outer timeout too. If that
timeout cuts an attempt, the timeout line says so.
- The wait loop keeps kubectl's stderr now. An expired credential or an
RBAC denial used to look the same as "not created yet" and ended in a
timeout on the object. Each error is printed once when it changes, and
the timeout line repeats the last one.
- When the linstor wait times out, the script lists every HelmRelease
first. The linstor release can be just waiting for a release further up
the dependency chain, and the Ready messages name which one.
- The MetalLB CRD wait and the CDI StorageProfile wait used to exit
silently. Now they print a timeout line and one last read of the missing
object with its error. The MetalLB wait also prints the metallb
HelmRelease.
- A failed or killed storage pool listing stops the script before any
pool is created. Before, it read as "no pools exist", and the creates
that followed failed with an unrelated error.

All of this is tested in `hack/post-install-prep.bats` with the existing
stub PATH and virtual clock. One test checks that every kubectl call on
the success path and on each timeout path runs under a bound. The
literal grep in `hack/linstor-storage-mode_test.bats` is replaced with a
test of the behaviour.

The rest of #3680 lives outside this script and moves to #4628: the
empty `{"error":{}}` in helm-controller's Helm log, previous-container
logs on passing runs, and node and pod resource state during install.

Closes #3708, closes #3707, closes #3713, closes #3714, closes #3680.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): bound every call in the post-install-prep step and make its failures name their cause, so a hung call fails the install instead of holding it until the job timeout
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Reliability**
* Cluster operations now have bounded wait times, with clearer
diagnostics when resources or endpoints are delayed or unavailable.
* Storage setup stops when pool listings fail instead of masking the
error.
* **Tests**
* Expanded coverage for timeout handling, error reporting, and failures
during storage and resource setup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 13:17:17 +03:00
Aleksei Sviridkin d8ec8f0af1 fix(ci): run every bats unit file after a failure and cover build-*.sh in the cross-compile check (#4633)
## What this PR does

Two fixes to the tooling of the unit lane.

A local run of the bats unit tests stopped at the first red file and
never ran the files after it, while the output looked like a finished
suite. Each bats file is its own make target, and without `--keep-going`
make stops scheduling targets after the first failure. CI passes `-k`,
so it was not affected. The target now runs the files in a
`--keep-going` sub-make. That sub-make still exits non-zero if any file
failed, and it shares the caller's `-j` slots and `--output-sync`. The
new `hack/bats-unit-keep-going.bats` drives the real Makefile against
two fixture files. It checks that both run and the target fails when the
first one fails, that the target exits 0 when both pass, and that the
sub-make still finds the root Makefile when `MAKEFILES` names another
file. I split the exact-line check on this target in
`hack/bats-runner-coverage.bats` in two to match the new recipe. A
workflow comment and a test name that still called each file a
prerequisite now call it a target.

The cross-compile check never looked at the multus-cni builder. It
treated only a script named exactly `*build.sh` as a compile step, and
multus runs upstream `./hack/build-go.sh`. The pattern now allows a
suffix after `build` but no `/`, and each case has a fixture test. Among
the Dockerfiles the check scans under `packages/`, this matches exactly
one new line, the multus stage, and that stage passes.

Closes #3692 for the make-level truncation. The in-file half, where
`hack/cozytest.sh` ends a file at its first failing test, is tracked in
#4620.

Closes #4500

Things I left out:

- The multus stage passes because a `curl ... TARGETARCH` step sits in
the same stage. The check does not see that `build-go.sh` reads
`TARGETPLATFORM`. The test header already documents this per-stage
limit.
- The sub-make reads only the root Makefile, so an extra `make -f
other.mk` given to the parent no longer reaches the bats targets.
Command-line variables and `MAKEFILES` still do. Nothing in the repo
calls it that way.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): `make bats-unit-tests` runs every bats file after a failing one and still fails at the end, and the image cross-compile check now covers build scripts such as `build-go.sh`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Unit tests now continue running across individual test-file failures
while still reporting an overall failure if any file fails.
* Build-stage detection now recognizes build scripts with names such as
`build-go.sh`.
* **Tests**
  * Added coverage for unit-test results and build-script detection.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 13:17:13 +03:00
Aleksei Sviridkin cdc4b88a90 fix(ci): close e2e suite selection gaps in select-e2e and select-install (#4634)
## What this PR does

The e2e test selector picked the wrong Chainsaw suites for some diffs.
In one case it lost an escalation, in another it ran the full suite
where a subset was enough, and in two more it missed a suite or ran
nothing. The fixes are in `hack/select-e2e.sh`, with the `.yml` part
also in `hack/select-install.sh`.

- A changed file in a directory under `hack/e2e-chainsaw/` that holds no
suite escalated only when it was the whole diff. Next to any path that
selected a suite, the escalation was dropped with nothing on stderr. The
directory name is now checked against the suite list right where the
path is classified, like the `hack/e2e-apps/` rule already did. The
empty-selection backstop can't be reached anymore. I kept it as a guard,
and the doc says its line has no test.
- A seaweedfs change ran the full suite, because
`cozystack.seaweedfs-application` mapped to a `seaweedfs` suite that
does not exist. Now it selects the suites that use tenant-root's object
storage: bucket, the backup round-trips that write through the
in-cluster `seaweedfs-s3` endpoint, and harbor, which keeps its registry
in a BucketClaim served by the seaweedfs COSI driver. The issue says
this mapping would break the round-trip test. It doesn't, since that
test only walks suite to source to suite, and `vm-disk-application`
already maps to `vminstance` the same way. The test takes the expected
set from the suites' own yaml.
- A suite written as `chainsaw-test.yml` was invisible to both
selectors. Chainsaw v0.2.15 loads that name when there is no
`chainsaw-test.yaml` (`pkg/discovery/load.go`), so such a suite runs but
never gets selected. Both scripts list it now. `select-install.sh
--validate` also passed when `find` failed, and now it fails.
- An edit under `examples/backups/_lib/` was read as an app named `_lib`
and selected nothing. It escalates now, same as
`hack/e2e-chainsaw/_lib/`.
- The selector comments quoted bats file counts and escalation-branch
counts. All of them were stale, so I removed them. The doc half of #4120
was already fixed on main.

Each fix has a test that fails on main. I also mutated each fix: its own
test goes red and no other test does.

Three related gaps outside these lines are tracked in #4630: the
`packages/core/testing/Makefile` comment that names only
`chainsaw-test.yaml`, suites built from numbered step files, and a file
placed directly in `examples/backups/`.

Two things worth knowing before merge:

- Any file except the `.disabled` one in a parked suite's directory,
like `hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml`, now always runs
the full suite. That is the safe direction. If it gets expensive, those
directories could be made inert.
- The `examples/backups/_lib/` rule and its doc line point at a
directory another PR adds. If this PR merges first, the rule just has
nothing to match yet.

Closes #3816, closes #3710, closes #4283, closes #4120, closes #4205,
closes #4626.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): e2e test selection no longer drops an escalation for a non-suite path under hack/e2e-chainsaw/, selects the suites that use seaweedfs instead of the full run, sees suites written as chainsaw-test.yml, and escalates edits to examples/backups/_lib/.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* End-to-end test selection now recognizes suites configured with either
supported YAML filename and reports discovery failures instead of
silently proceeding.
* Changes to shared backup walkthrough helpers and unrecognized
test-suite directories now trigger broader test coverage.
* Storage-related changes in SeaweedFS now select all relevant test
suites.

* **Documentation**
* Updated testing guidance to describe suite discovery, file-change
matching, and when the full test suite is selected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 13:16:55 +03:00
scooby87 031ded71ee feat(seaweedfs): alert before an instance stops taking writes (#4573)
## What this PR does

A SeaweedFS instance that runs out of room stops taking writes without
anything paging. A volume server below its `minFreeSpacePercent` reserve
turns every volume it holds read-only; before that, it can run out of
slots, and a bucket whose volumes are all full then cannot get a new
one, even with half the disk free. Backups archiving into the instance
fail in turn, and a Postgres cluster doing continuous WAL archiving
keeps the segments on its own volume until that fills too.

The generic `KubePersistentVolumeFillingUp` alert does not catch this:
it pages below 3% free, while a volume server stalls at its 5% reserve
or, when slots run out, with the disk still half empty; and its early
warning needs the disk to keep filling, which stops once writes fail.

This PR adds per-instance rules to the SeaweedFS system chart, on
signals the servers already publish and the platform already scrapes,
rendered under the same `monitoring.enabled` gate as the chart's
ServiceMonitors:

- `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less
than 10% of its disk left above the reserve.
- `SeaweedFSVolumeServerFull` (warning): a volume server cannot host
another volume, because it is at the reserve or has no slot left.
- `SeaweedFSWritesFailing` (critical): the master has failed to place a
write in every 5-minute window for 15 minutes. Writes may still land in
between; the usual cause is a bucket with no writable volume left that
the master cannot grow. A new bucket's first writes fail a few picks
while its volumes grow, which the 15-minute hold does not reach unless
new buckets keep arriving.

Every expression is pinned to the release namespace, because the root
vmalert evaluates each instance's rules against every namespace's
metrics.

It also adds `docs/operations/seaweedfs-capacity-for-backups.md`: how to
size an instance for continuous WAL archiving, what the rules mean, and
how to grow an instance.

The expressions were validated live rather than by unit tests, since the
repository has no rule-evaluation tooling: on a test cluster, a
standalone SeaweedFS 4.45 on a 1 GiB volume was scraped by the
platform's vmagent and the rules rendered from this chart were evaluated
by the root vmalert. All three loaded healthy and each reached `firing`:
`SeaweedFSVolumeServerDiskLow` with 5.7% of the disk left above the
reserve, `SeaweedFSVolumeServerFull` both at the reserve and on slot
exhaustion, which stopped one bucket at 50% disk use, and
`SeaweedFSWritesFailing` under continuous failing writes, resolving once
they stopped. A new bucket's first successful writes on a free disk did
fail a few picks, which is why that rule holds for 15 minutes. On the
healthy platform instance all three stay quiet. helm-unittest pins what
a render can break: the namespace selector of each rule, the shared
label set of the two arms of the full-server rule, the gate, the
critical rule's hold, and that alert templating reaches vmalert
unrendered; each assertion was checked to fail when its line is mutated.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches a system chart template, its test and a document under
`docs/operations/`; no app values, platform values,
ApplicationDefinition, tooling or `hack/` path from the trigger map.

### Release note

```release-note
feat(seaweedfs): alert when a SeaweedFS volume server nears its free-space reserve, cannot host another volume, or keeps failing to place writes, and document how to size an instance for continuous WAL archiving.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added namespace-scoped SeaweedFS monitoring alerts for low or
exhausted disk capacity, unavailable volume slots, and write-placement
failures. Capacity alerts use a 10% disk threshold, and alerts trigger
after 15 minutes; write failures are marked critical.
- **Documentation**
- Added guidance on estimating SeaweedFS storage needs for Postgres
backups, interpreting capacity alerts, and expanding storage when
supported by the StorageClass.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 10:29:35 +03:00
Aleksei Sviridkin 97fc4359ad feat(scheduler): set scheduling-class label on mutated pods (#2621)
## What this PR does

Extends the lineage-controller-webhook so that, on every Pod it mutates
with a `SchedulingClass`, it also sets a label of the same name (in
addition to the existing annotation).

Today, when a SchedulingClass is in effect on a tenant, the webhook
injects:
- `spec.schedulerName: cozystack-scheduler`
- annotation `scheduler.cozystack.io/scheduling-class: <name>`

This is enough for the cozystack-scheduler to look the class up and
apply its `nodeAffinity`, `nodeSelector`, and topology rules. However,
it is **not** enough to write a `SchedulingClass` whose
`spec.podAffinity` rules `labelSelector` targets pods across multiple
applications, because `labelSelector` matches labels and not
annotations.

This PR adds the label, leaving the annotation untouched for backward
compatibility:

```go
podLabels := obj.GetLabels()
if podLabels == nil {
    podLabels = make(map[string]string)
}
podLabels[schedulerapi.SchedulingClassLabel] = schedulingClass
obj.SetLabels(podLabels)
```

The constants `SchedulingClassLabel` and `SchedulingClassAnnotation`
exported by `github.com/cozystack/cozystack-scheduler/pkg/apis/v1alpha1`
already share the same value
(`scheduler.cozystack.io/scheduling-class`), so no new key is
introduced.

**Why this is useful**: a single SchedulingClass can now express
"co-locate every pod that uses me at the same topology" with a rule
like:

```yaml
apiVersion: cozystack.io/v1alpha1
kind: SchedulingClass
metadata:
  name: co-region-prod
spec:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          scheduler.cozystack.io/scheduling-class: co-region-prod
      topologyKey: topology.kubernetes.io/region
```

When several applications share this class, the second-and-later pods
will be constrained to the region the first one landed on — without
hard-coding any region in the class itself.

### Tests

Adds a `webhook_test.go` (none existed in the package). Five unit tests,
using `clientfake.NewClientBuilder` +
`dynamicfake.NewSimpleDynamicClientWithCustomListKinds`, cover:

- Label, annotation, and `schedulerName` are all set when a class
exists.
- Existing pod labels are preserved (no clobbering).
- Non-Pod objects are a no-op.
- Pods in a namespace without a scheduling-class label are a no-op.
- Pods whose referenced `SchedulingClass` CR does not exist are a no-op
(the existing defensive behavior, preserved).

### Release note

```release-note
feat(scheduler): the lineage-controller-webhook now sets a scheduler.cozystack.io/scheduling-class label on mutated pods (in addition to the existing annotation), enabling SchedulingClass authors to write podAffinity rules that match pods across applications.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Pods with a valid scheduling class now receive its label alongside the
scheduler configuration and annotation.
- Existing Pod labels are preserved when the scheduling-class label is
added.
- **Bug Fixes**
- Scheduling-class settings are applied only when the namespace
references a scheduling class that exists. Non-Pods and namespaces
without a scheduling-class label are left unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-01 08:58:42 +03:00
mattia-eleuteri 250f89dc72 feat(scheduler): set scheduling-class label on mutated pods
The lineage-controller-webhook now sets a
scheduler.cozystack.io/scheduling-class label on mutated pods (in
addition to the existing annotation of the same name).

This unlocks SchedulingClass authors writing podAffinity rules whose
labelSelector targets pods across different applications that share
the same class. A natural pattern becomes: define a class whose name
is the affinity-group identifier, attach it to several applications,
and have the scheduler co-locate them at the chosen topology key
(region, zone, hostname).

Behavior preserved when the SchedulingClass CR is missing: pods are
left untouched (no scheduler injection, no label) so they do not stay
Pending on a non-existent scheduler.

Tests cover label/annotation/schedulerName injection, existing label
preservation, non-Pod no-op, no-class no-op, and missing-CR no-op.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-10-01 05:59:24 +03:00
Aleksei Sviridkin 1e81164d0e fix(kubevirt): route the export proxy to the Service KubeVirt creates
virt-operator names the export proxy Service virt-exportproxy. The
Ingress and the TLSRoute publishing vm-exportproxy.<host> named a
vm-exportproxy Service instead, which nothing creates, so both routes
had no backend. The public hostname stays vm-exportproxy.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 03:11:22 +03:00
Aleksei Sviridkin 873d9f4c8e fix(backups): stop the nats and kafka-metadata demos using a Pod they refused
Both demos read their CLI output through command substitutions, where
bash does not apply errexit. When the ownership check refused a
same-named Pod another party owns, the refusal was ignored and the exec
went into that very Pod. A failed read of the Pod's phase or owner was
likewise read as "no Pod", so the helper went on to delete and replace
a Pod whose owner it had not seen.

Every step of the Pod setup now returns on failure, and the phase and
owner reads tell a missing Pod (--ignore-not-found) from a failed read.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-01 02:21:01 +03:00