A failed DataVolume List keeps the monitor's last verdict and retries,
and the only trace of it was a controller log line. A monitor with no
verdict yet then reads operational while its disks are unreadable, with
nothing a tenant can see saying why. A Warning event on the monitor
makes the failure visible where the bucket metrics failure already is.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A controller-runtime Kind source adds an event handler to the informer
when it starts and never removes it; cancelling it on a sync timeout
only ends its context. Building a new source on every failed sync
attempt therefore left one more handler on the DataVolume informer per
retry, each enqueueing the same events, until the controller restarted.
The source is now started once and later attempts only wait for the
informer to sync. Because the started source keeps delivering events
between attempts, the syncing flag stays set until the reader is
installed, so a reconcile queued in that gap is requeued as well.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Starting the DataVolume source replays every DataVolume to its
handler at once, and the handler queues the matching monitors, while
the reader is installed only after the source has synced. A monitor
reconciled in that window saw no reader, kept its last verdict and
returned with nothing left to queue it again, so a disk that failed
or finished while the controller was down could keep a wrong
WorkloadsReady until the next unrelated event.
Such a reconcile still publishes the replica counts and now asks to
be requeued after a few seconds, until the reader is in place or the
sync attempt gives up. It is a requeue rather than an error because
the window is expected: an error would count as a reconcile failure
and log a line for every monitor in the cluster while it lasts.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ready follows the Helm release; whether CDI finished is in
WorkloadsReady. Say so and where an operator sees it, say that a
disk created before this change keeps a release that may wait on the
DataVolume until its next update, say that an upload disk reads not
ready until the upload completes, and say that a disk waiting for its
first consumer counts as ready, so it is not read as broken.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The console badge followed Ready alone, and Ready follows the Helm
release. A vm-disk install does not wait on its DataVolume, so a disk
CDI never populated was painted green.
The badge now shows the WorkloadsReady condition while Ready is True
and WorkloadsReady is not True with reason DataVolumeNotReady. Other
causes of WorkloadsReady=False keep the badge as it was: a stopped VM
has no pod and reports not operational on purpose, and turning it
amber for every kind needs its own decision. A failing Ready still
wins, because it is the more specific cause.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
WorkloadsReady=False covers different situations: a stopped VM has
no pod and its monitor reports it not operational on purpose, while a
disk whose DataVolume never populates is broken. A consumer that wants
to react to one and not the other had only the message text to go by.
The WorkloadMonitor status gains a Reason next to Message, set to
DataVolumeNotReady by the DataVolume check, and the Application API
carries it into the WorkloadsReady condition in place of the generic
reason whenever a monitor that is not operational names one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ready follows the Helm release, so for a kind whose install does not
wait on its workloads, a vm-disk among them, READY in kubectl get says
only that the manifests were applied. Whether the workloads came up
lives in WorkloadsReady, which the default table did not show, so a
disk CDI never populated looked healthy on the surface an operator
reads first.
The table now carries a WORKLOADS column beside READY. It reads
<none> for an Application with no WorkloadMonitor, which has no
WorkloadsReady condition at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The DataVolume List sits on the reconcile path of every
WorkloadMonitor, and returning its error before the status write froze
the replica counts and Operational of a monitor that selects no
DataVolume at all whenever CDI's API hiccupped.
The status is now written either way, with the last DataVolume verdict
kept whenever the DataVolumes could not be read, and the error still
returned so the request is retried. The same applies while no reader
is installed yet, so a stuck disk does not read as ready for the
window between a controller restart and the DataVolume informer
syncing.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
For a vm-disk, WorkloadsReady is the one condition that reports
whether CDI populated the disk, and its generic "One or more workloads
are not operational" gives an operator nothing to act on: a disk has
no workloads, and the stuck DataVolume and its phase are exactly what
they need to know.
The WorkloadMonitor status gains a message that the controller fills
with each selected DataVolume that is not ready and its phase, and the
Application API uses the messages of the monitors that are not
operational in WorkloadsReady instead of the generic text whenever one
of them names a cause.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The VMDisk scenarios waited on the HelmRelease, which does not cover
the import behind it. They now wait for the DataVolume to be Ready and
then for Ready and WorkloadsReady on the VMDisk itself. The DataVolume
wait keeps the combined budget the two old waits had.
That alone would pass with the monitor ignoring DataVolumes, since an
empty monitor reports operational. A new scenario imports from an
unresolvable URL and waits for Ready True with WorkloadsReady False,
which only a monitor that reads the DataVolume phase can produce. A
DataVolume that has not started is not ready either, so the scenario
first waits for the importer to fail, which CDI reports as Running
False with reason Error, and only then asserts the VMDisk.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm's default poller falls back to a resource's Ready condition for
kinds it has no rule for, so a vm-disk install could wait on the
DataVolume's Ready and hold a helm-controller worker for the whole
install timeout while an import runs or never completes. Whether CDI
populated the disk is reported by the WorkloadMonitor, so the release
now treats the DataVolume as current and Ready of a vm-disk no longer
depends on the import. Release settings reach a HelmRelease when its
application is created or updated; the reconciler that syncs
definitions onto existing releases copies the chart reference,
valuesFrom and labels only, so an existing disk picks this up on its
next update.
The vm-disk-rd package had no unit test target; it gets one, and a
test that pins the entry.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A vm-disk application reported its workloads operational whatever
state its DataVolume was in, so an import or clone that never
finished left the disk looking healthy while it was unusable. Its
monitor has minReplicas 0 and only pods fed Operational.
The monitor now also lists the DataVolumes its selector matches and
reports not operational while any of them is in flight, Failed,
Unknown or without a phase. UploadReady counts as in flight: CDI
holds it from the moment the upload server is ready until the
transfer succeeds, so it covers an empty disk and a half-uploaded
one alike. Waiting for the first consumer counts as ready, since
that is where such a disk rests, and so does a phase this
controller does not know, so a CDI release that renames one cannot
turn working disks red.
This is deliberately not a Helm wait: a readiness gate holds a
helm-controller worker for its whole window, and a disk that never
populates would keep taking it back. The signal lands in the
application's WorkloadsReady condition instead, and Ready keeps
following the HelmRelease.
CDI ships with the IaaS bundle only and can be installed while this
controller is running. A watch on a kind whose CRD is absent keeps
the controller from starting, so the DataVolume watch is added from a
runnable that polls discovery until the kind is served. It starts on
the first reconcile, the earliest point at which a new watch is
started at once, rather than on the next tick, so a disk created
just after a restart is not read as populated for a poll interval.
The reader is installed only once that watch has synced within a
bound: a reconcile context has no deadline, and a List on an
informer that never syncs would park every WorkloadMonitor reconcile
for good.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Two small bug fixes, one per commit. Both were first proposed by
@officialasishkumar in #2334 and #2336. Those PRs went stale, and part
of each was fixed on main in the meantime, so this carries only what is
still broken.
ClickHouse Keeper hosts rendered without a cluster domain, as
`chk-<name>-keeper-cluster1-0-0.<namespace>.svc.` with nothing after
`svc.`. The zookeeper block in `clickhouse.yaml` redeclared
`$clusterDomain` from `.Values.clusterDomain`, which the chart never
defines, and that shadowed the value read from `_cluster.cluster-domain`
at the top of the template. Keeper is enabled by default, so every
ClickHouse release got these hosts. The keeper name test set
`clusterDomain` itself and hid the bug. It now sets only
`_cluster.cluster-domain`, like the platform does, and fails without the
fix.
The WorkloadMonitor controller panicked on a PVC with no
`storageClassName`, which is any PVC that takes the cluster default
class. The check used `||` where it needed `&&`. controller-runtime
recovers the panic, but the reconcile of that monitor fails on every
retry and never reaches its status update. A PVC with an empty
`storageClassName` also got a resource key with an empty class name
instead of `default`. A new table test covers unset, empty and explicit
classes.
The duplicate `template` key in the ClickHouse backup CronJob from #2334
and the `Operational` status fix from #2336 are already on main, so they
are not in this PR.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(clickhouse, workloadmonitor): ClickHouse Keeper node hostnames now include the cluster domain, and the WorkloadMonitor controller no longer panics on a PVC without a storage class.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Workload resource labels now use the default storage class when a
PVC’s storage class is unset or empty, while continuing to reflect
explicitly configured storage classes.
* ClickHouse ZooKeeper hostnames now consistently use the configured
cluster domain, or `cozy.local` when no domain is set. This helps
hostnames align with the cluster’s domain configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Stacked on #4468, which holds the raised backup waits to a start margin;
review that one first. This PR's own diff is the six commits on top of
it.
Most HR-Ready waits in the E2E suites gave up before the release they
were waiting on could fail its install. cozystack-api gives every
release it generates a 600s install timeout, and the Chainsaw asserts in
front of those releases sat at 5 or 6 minutes. When an install genuinely
failed, the suite reported a bare assert timeout against a release that
was still installing, and the reason the install failed, which the
release records when its own timeout fires, was never reached. Red runs
failed just as often as they should; they only said less.
I raised every HR-Ready wait above the install timeout of its own
release, by the 60s start margin #4468 introduces:
- Chainsaw asserts on releases cozystack-api generates go to 11m, across
bucket, clickhouse, etcd, external-dns, foundationdb, harbor, mongodb,
openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and
vminstance.
- Asserts on nested `<name>-system` releases take their floor from the
template that renders them: 11m for external-dns, openbao and qdrant
(10m `spec.timeout`), 16m for `harbor-test-system` (15m). `harbor-test`
sat exactly on its 600s floor, which expires first because the assert's
clock starts before the install's; it goes to 11m too.
- The kuberture Package release takes its 10m from cozystack-operator
and goes to 11m.
- In the backup walkthroughs, the bucket waits go from 300s to 660s in
clickhouse, kafka, mariadb, mongodb, postgres, rabbitmq and redis, and
the two postgres waits go from 360s to 660s. The kafka bucket wait
passed no budget and silently took the helper's 300s default, which is
why earlier scans for literal budgets missed it. Each Chainsaw op that
runs a walkthrough grows by exactly what its waits gained (clickhouse
78m, kafka 46m, mariadb 48m, mongodb 42m, postgres 66m, rabbitmq 48m,
redis 36m), so a failure at any later wait keeps the headroom it had and
is still reported by the script instead of being cut off by the op.
Two waits stay as they are, on purpose:
- kafka: `kafka-rd` sets a 7h install timeout, which no budget inside
the 215-minute E2E job can clear. The kafka assert and the Kafka waits
in the kafka and kafka-metadata walkthroughs keep their numbers, and the
assert now says why in-line.
- `vm-instance-test`: `vm-instance-rd` disables the install wait, so its
release goes Ready as soon as helm has applied and there is no floor to
clear. Earlier lists of this problem included it by mistake.
The cost falls on red runs only: a suite whose release never goes Ready
now spends up to six more minutes before it reports, and in exchange the
report carries the release's own reason. Green runs are unchanged.
Suites run one after another in a single job capped at 215 minutes, so a
systemic failure that reddens many suites now reaches that cap sooner,
and a job cancelled at the cap loses its log and `cozyreport.tgz`. I
took that trade because a failed install that says why is worth more
than a red run that ends a few minutes earlier.
`docs/agents/e2e-testing.md` stopped being true in several places once
the waits were raised: it described most of the tree as under the floor,
mariadb as the only suite raised, the walkthrough bucket waits as under
it, and harbor's asserts as sitting on and under theirs. I rewrote those
sentences, and the one that pointed at "every wait above the band" of
5-6m, kept the start-margin wording from #4468, and added that a
walkthrough wait passing no budget takes the helper's default. The
comments in `hack/mariadb-first-boot-waits.bats` that quoted the old op
and bucket figures are restated; its existing checks and constants are
unchanged, because every op grew by exactly its bucket delta and each
remainder still sits at exactly its constant. I also added postgres to
the table that file holds the walkthroughs to, so its raised waits and
its op cannot drift back apart unnoticed, and a test that holds each
table suite's bucket wait to the bucket release's floor, read from the
api-server default with `bucket-rd` checked for an override; the op
check alone could not catch a bucket wait dropping back, since a lower
wait only widens the slack. redis stays out: its walkthrough waits on no
redis HelmRelease, only on the bucket and on the sentinel master, so the
table's floor check has nothing to read there.
How I checked it, since E2E cannot run locally:
- I re-ran the enumeration from the sizing section of
`docs/agents/e2e-testing.md` over every
`hack/e2e-chainsaw/*/chainsaw-test.yaml` and every `wait_hr_ready` call
under `examples/backups/`, literal or default, and held each against its
own release's floor. Every `Ready=True` assert and every `wait_hr_ready`
call a suite drives now clears its floor, except the kafka ones named
above. The opt-in etcd round-trip uses bare `kubectl wait hr` calls at
300s instead of that helper; they are listed in #4470. The `Ready=False`
assert on the rejected redis upgrade is not a readiness wait and is left
out.
- `make bats-unit-mariadb-first-boot-waits` passes, 31 of 31. For the
new postgres row, putting the op back to 50m fails only the postgres
slack test, and a postgres wait at 659s fails only the postgres floor
test. Setting any one table suite's bucket wait to 659s fails only the
bucket test, naming only that suite, and an install-timeout override on
`bucket-rd` fails it too. I ran the mutations under plain `bats` so a
fail-fast runner could not hide a second failure.
- `hack/select-e2e.sh` over this diff selects bucket, clickhouse, etcd,
external-dns, foundationdb, harbor, kafka, kuberture, mariadb, mongodb,
openbao, opensearch, postgres, qdrant, rabbitmq, redis, valkey and
vminstance.
Out of scope, and tracked in #4470: the tenant addon waits in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` (an edit there escalates the
whole E2E run), the redis and kafka-metadata `wait_hr_ready` helpers
that print no reason on timeout, and the opt-in etcd round-trip's bare
`kubectl wait` calls.
Known and left for later, none of them changed here:
- Only the mariadb Chainsaw asserts and the walkthroughs in the guard's
table are held by a test. The other raised asserts, and the redis and
kafka bucket waits, rest on literals with in-line reasons; #4470 covers
extending the guard.
- The Kafka waits in `examples/backups/kafka/04-create-kafka.sh`,
`07-restore-to-copy.sh` and `kafka-metadata/run-all.sh` stay at 300s
against the 7h floor without an in-line reason at the call site. The
doc's rule asks that of Chainsaw asserts only, and the kafka Chainsaw
assert carries it.
The postgres waits are the ones #3748 is about. Their budget now clears
the install-timeout floor and a test holds it there; what a CNPG first
boot costs on this platform is still unmeasured, so that issue stays
open.
Closes#3771Closes#3753
### Screenshots
Not a UI change.
### Downstream repositories
I walked the trigger map in `docs/agents/contributing.md` against the
diff. It touches E2E suites, backup walkthrough scripts, a unit-lane
bats file and an agent doc, none of which any downstream repository
restates.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(tests): HelmRelease readiness waits in the E2E suites and backup walkthroughs now outlast the install timeout of the release they wait on, so a failed install reports its own reason instead of a bare timeout.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Extended readiness and backup round-trip test timeouts across multiple
services to accommodate slower startup and provisioning.
* Added checks to verify that bucket readiness waits meet configured
installation-timeout requirements.
* **Documentation**
* Updated testing guidance to clarify how readiness waits relate to
release-specific timeouts and overall test budgets.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The isp-hosted variants install no CNI of their own, so on a host
without Cilium the cilium.io/v2 kinds do not exist and the tenant
install fails, taking the root tenant and everything that depends on
cozystack-basics down with it.
The policies now render on isp-hosted only when the host serves both
Cilium policy kinds. On every other variant Cozystack installs Cilium
itself, so they keep rendering unconditionally: a CRD that is missing
at render time there fails the install loudly instead of leaving the
tenant without network isolation, because helm-controller does not
re-render a release when only API discovery changes.
On isp-hosted without Cilium the tenant gets no network isolation from
Cozystack; isolating tenants is then up to the host's CNI.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
PUSHED_TAGS_LOG fed every tag to hack/stitch-multiarch.sh whether or
not the build pushed it, so a PUSH=0 build, and now a plain LOAD=1
build, handed the stitch tags that were never published.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
image-talos publishes with skopeo instead of buildx, so it never saw
--push=$(PUSH) and a local PUSH=0 build still overwrote talos tags in
REGISTRY.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A plain := assignment beats an environment variable, so
`LOAD=1 PUSH=0 make image` pushed a multi-arch build instead of
loading one. An empty value from the environment falls back to the
default rather than reaching buildx as `--push=`. PUSH now also
defaults to 0 under LOAD=1, so a local load never publishes unless
PUSH=1 is asked for. The OCI_EXPORT_DIR block still overrides an
environment PUSH or LOAD.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
No caller passes the second argument any more, and an explicit tag
would bypass CACHE_TAG, so the amd64 and arm64 legs of such an image
would write the same mode=max cache ref and evict each other's layers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Every flux install fetched fluxcd.io/install.sh and ran it as root, so
whatever that host served at the moment executed on runners that later
hold registry credentials. Use the upstream setup action instead, pinned
by commit and given an explicit version: it checks the release tarball
against that release's checksums file and runs no script fetched at run
time.
nightly and the fork e2e lane still took yq from the floating latest
release. They now use the pinned, checksum-verified install the
promotion steps already have.
The helm-unittest plugin embeds its own Helm and was installed at
whatever version was newest, so a plugin release could change chart
test results on a PR that touched no chart. Pin it, and replace a
different version a persistent runner may carry.
A contract test scans every workflow for these three tools, so an
unpinned flux, yq or helm-unittest install copied from an older step
fails the unit lane.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds the opt-in shared-central traces topology from the distributed
tracing design (cozystack/community
`design-proposals/distributed-tracing`, rollout item 4): a tenant can
send its spans to one VictoriaTraces store in tenant-root instead of
running its own, with both write and read isolation between the tenants
that share it.
**Switching it on.** A tenant opts in with the new Tenant field
`tracingCentral`, which needs the tenant's own Monitoring. The tenant
chart publishes it to the tenant's monitoring charts as
`_namespace.tracingCentral` in the `cozystack-values` Secret, which the
apps API does not let an application write, and renders the two network
paths the topology needs under the same condition. tenant-root decides
whether it hosts the store, with the new Monitoring field
`tracingCentralHost` next to a cluster-mode `tracingStorages` entry
named `generic`.
**Isolation.** VictoriaTraces takes the tenant account from the
`AccountID`/`ProjectID` headers and does no authorization of its own, so
other tenants reach the shared store only through a vmauth in
tenant-root. Each central tenant gets a VMUser that pins an account
derived from its namespace name and UID, for both its collector's writes
and its Grafana reads, and vmauth overrides any account a client sends.
The collector and Grafana log in with credentials the chart generates
and keeps. The tenant egress rules allow only pods carrying the
collector's or Grafana's label to reach the vmauth proxy port, and a pod
that carries such a label without the tenant's credentials gets only 401
from it; the vmauth internal routes, whose `/metrics` lists every
tenant's user, are served on a port tenants cannot reach. The
tenant-root namespace itself is inside the trust boundary, as it already
is for the metrics of tenants that rely on a parent's monitoring: its
workloads reach the store directly, and one that learns a tenant's
account ID can read and write that account, and whoever tenant-root
admits to its Grafana reads the store through a datasource that bypasses
vmauth (with the System oidc mode, tenant-root's own groups, which
already hold the same access level in every tenant under it); the
`tracingCentralHost` description says so.
**Sharing the store.** Each tenant's VMUser is capped at four requests
in flight per vmauth replica, reads and writes together, and each
collector replica exports with two senders, so more collector replicas
do not raise a tenant's throughput; above the cap vmauth queues for up
to 10s and answers 429, which the exporter retries. The store has one
disk and one retention for all of its tenants and nothing reserves space
per tenant; `tracingCentralHost` states that, since hosting is
tenant-root's decision. The vmauth runs two replicas spread across nodes
under a PodDisruptionBudget, so draining one node does not stop every
central tenant at once. Every traces store is also disk-bounded now, as
the design requires: without `retentionDiskUsageBytes` it drops its
oldest day once its filesystem is 80% full. That alone does not keep one
tenant from filling the shared disk, since VictoriaTraces always keeps
the last two days.
**States that would drop spans silently are refused, one race aside.** A
live render fails, with a message naming the prerequisite, when a tenant
turns central on while tenant-root hosts nothing, and when tenant-root
stops hosting while central tenants remain; tenant-root's render also
refuses `tracingCentralHost` without a store to host. The second check
does not wait on a VMUser that is already being deleted, since its
tenant is leaving. Each check reads the other side's objects when it
renders, so a tenant turning central on while tenant-root stops hosting
can pass both and export to a vmauth that is gone; the
`tracingCentralHost` description names this race and its recovery. At
runtime, tenant-root's Monitoring release is not Ready while the vmauth
has failed. These checks run on a render, so removing the tenant-root
Monitoring altogether is not refused, and from then on every Monitoring
render of a central tenant fails until it turns central off; the
`tracingCentralHost` description says to turn central off on the tenants
first. Turning central on keeps the tenant's own local stores and their
datasources, so traces stored before the switch stay readable until the
tenant removes them; turning it off again leaves the spans written
meanwhile in the shared store, unreadable from the tenant until it turns
central on again, and spans sent while the collector switches back may
be dropped; a deleted tenant's spans likewise stay until retention drops
them. Right after a tenant turns central on, or after its credentials
Secret is re-created, spans its collector sends before the vmauth has
loaded the new login get a 401, which the exporter does not retry, so
they are dropped; the tracing guide in #3772 covers it.
**Where this departs from the design.** The design has the per-tenant
collector inject the account headers and adds exactly one egress rule.
Here the account is pinned by the tenant's VMUser at the vmauth and the
collector sends none, because the collector runs in the tenant namespace
and whatever runs with its pod label could otherwise choose an account;
vmauth is also what the design already requires for reads, so both
directions go through it. That needs a second egress rule, for the
tenant's Grafana to reach the vmauth, beside the collector's. The
per-tenant write rate the design gives the collector is not in this PR:
no OpenTelemetry distribution has a rate-limiting processor, so #4648
builds the collector with one and holds each central tenant to a rate
tenant-root publishes. Until it lands, a tenant writing faster than the
disk cap allows over two days can fill the disk the other tenants share.
**Consent is per host, not per tenant.** `tracingCentralHost` lets every
tenant that sets `tracingCentral` in, nested tenants included, and each
gets its own VMUser and its own request cap; there is no allow-list. A
tenant-root that needs to limit who shares its store keeps
`tracingCentralHost` off.
**Upgrade.** Tenants that do not opt in render the same objects as
before, and their `cozystack-values` Secret is unchanged. The monitoring
charts render the same objects too, with one exception: a VTSingle or
VTCluster without `retentionDiskUsageBytes` gains
`retention.maxDiskUsagePercent: "80"` in its `extraArgs`, so its spec
changes and its storage pods restart once, and a store already more than
80% full drops its oldest days when it starts. The inner monitoring
HelmRelease's values gain the new `tracingCentralHost: false` default,
as with any new field, and its `healthCheckExprs` gain a `VMAuth` entry,
which matches nothing outside a hosting tenant-root. Both change the
HelmRelease spec, so every tenant's monitoring release upgrades once,
rendering the same objects. A tenant-root that has a cluster-mode
`generic` store gets no vmauth until it sets `tracingCentralHost`.
**Testing.** helm-unittest covers the three charts the change touches
(tenant, monitoring, extra monitoring) and every guard was checked by
mutation; a bats test covers what helm-unittest cannot, that the
generated password and the collector's rollout checksum agree on a first
install. On a test cluster: a spoofed `AccountID` from a tenant is
overridden by vmauth, one tenant cannot read another's spans, the vmauth
starts under the real tenant network policies (and stays in init without
its apiserver label), the guards refuse as described, and the rendered
collector config passes `otelcol validate` in the pinned image. There is
no end-to-end test yet; one that covers cross-tenant isolation is
planned as a follow-up.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: covered by cozystack/cozystack#3772, the tracing guide
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#64
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
The Tenant gains the user-facing field `tracingCentral` (bool, default
false), which the trigger map sends to terraform-provider-cozystack: its
hand-written Tenant schema and model need the field. The follow-up is
cozystack/terraform-provider-cozystack#64. The Monitoring gains
`tracingCentralHost` (bool, default false); the provider's
`cozystack_tenant_module` is a marker without values, so it needs no
change. The website reference pages for both are generated from the
READMEs on release. The tracing user guide in #3772 covers the
per-tenant backend only. A central-mode section in the tracing guide of
#3772 (hosting and opting in, the shared disk and retention, turning
central off on the tenants before tenant-root stops hosting, and the
recovery from the opt-in/stop-hosting race) also takes the lifecycle
details the `tracingCentral` description no longer carries.
### Release note
```release-note
feat(monitoring,tenant): add an opt-in shared-central traces store. A tenant sets tracingCentral to send its spans to one VictoriaTraces store in tenant-root instead of its own, and tenant-root opts in to hosting it with tracingCentralHost on its Monitoring. Tenants sharing the store are isolated on write and read through a vmauth in tenant-root, and each is capped in the requests it can hold in flight.
```
## What this PR does
This PR turns off monitoring instance of root tenant in e2e install and
removes waits on it. Install patched root tenant with `monitoring: true`
and then waited up to 15m each on vmalert and vmalertmanager,
`vlclusters/generic`, shortterm and longterm vmclusters, `grafana-db`
CNPG cluster and grafana deployment, but nothing in the run looks at any
of it.
No chainsaw suite touches tenant-root monitoring objects. Only one that
mentions monitoring is `foundationdb`, it asserts `WorkloadMonitor`
whose CRD comes from `cozystack-controller` and exists without
monitoring stack. Monitoring OIDC wiring is covered by
`packages/system/monitoring/tests/oidc_test.yaml` since #4019 removed
orphaned BATS suites. Kubernetes suites run with
`monitoringAgents.enabled: false` and tracing is off by default
(`tracingStorages` is empty), so they don't depend on this instance
either.
Those waits were not free. Monitoring stack competes with rest of
install for node capacity, and removed block own comment said slow
victorialogs start can fail install on PR that never touched monitoring
- that's why it was moved to 15m budget.
### What stays
Monitoring package stays installed, because `tenant-application` has
`dependsOn cozystack.monitoring-application` and without the package
platform release is never resolved. Only tenant-root instance is off.
Container lane `logsStorages` override for victorialogs also stays. It
does nothing while monitoring is off, but keeps a run that turns
monitoring back on from creating immutable replicated PVC on lane that
has only `local`. Its comment, paragraph in `docs/agents/e2e-testing.md`
and comment on Install step env in `pull-requests.yaml` now say that
instead of calling root monitoring install-time workload.
### What is lost
`curl` through `root-ingress-controller` with `Host:
grafana.example.org`, it was the only check that root ingress actually
routes to backend. Grafana was the target just because it was installed.
Moving it to seaweedfs host needs expected response checked on live
stand first, so it's tracked in #4615.
### Savings
Not measured. The happy path never spent the 15m budgets, so what this
returns is real bring-up time of the stack plus tail risk, and only CI
runs can show how much it is.
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
NONE
```
## What this PR does
Removes e2e asserts that wait for something an earlier assert in the
same test already proves. First step of
[#2653](https://github.com/cozystack/cozystack/issues/2653), not the
whole issue.
qdrant and external-dns render the workload in a nested `-system`
HelmRelease, and both suites already assert that HelmRelease Ready. The
`readyReplicas: 1` on the StatefulSet and the Deployments goes, the
instance-label pin stays because the catch collectors select on it. I
checked on a dev cluster that a parent HelmRelease waits for its nested
one (a Qdrant with a broken `storageClass` kept the parent in
`Progressing` until its install timeout, then it failed naming the
nested HelmRelease), details in [this
comment](https://github.com/cozystack/cozystack/issues/2653#issuecomment-5929644802).
vminstance: both PVC `Bound` asserts follow a DataVolume `Ready` assert,
and a DataVolume is not Ready before its PVC is Bound.
Left alone on purpose: the qdrant PVC assert (the reclaim-on-delete step
only proves anything if the PVC was seen first, same in kafka), openbao
sts and PVC (they run after unseal), operator-created workloads, and the
Service and Endpoints shape checks. Chainsaw isn't installed on my
machine, so the suites are checked by CI only.
### Screenshots
No UI changes.
### Downstream repositories
The diff only edits `hack/e2e-chainsaw/*/chainsaw-test.yaml`. The
trigger map's `hack/` entry is for cozystack/ccp, whose skills anchor on
`hack/package.mk`, `hack/common-envs.mk` and `make generate`, none of
which change here. No box ticked.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
test(e2e): the qdrant, external-dns and vminstance suites no longer repeat readiness checks that the HelmRelease Ready assert already covers.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Updated end-to-end checks to use Helm release readiness for Deployment
and StatefulSet scenarios, rather than separately asserting ready
replica counts.
* Removed the requirement for PersistentVolumeClaims to reach `Bound` in
the VMDisk and VMInstance scenarios; DataVolume readiness checks remain.
* Retained instance-label checks and clarified that PVC reclaim
validation requires the claim to be observed first.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
This fixes two cleanup failures in the kubernetes suites. The nightly
runs failed `kubernetes-previous` and `kubernetes-latest` in the
replicated storage lane. PR runs fail `kubernetes-latest` with `LINSTOR
pools did not return to their pre-suite baseline within 300s`.
The replicated lane asked for too much free space at cleanup. It wanted
90 GiB free on every LINSTOR node, and volumes from earlier suites kept
srv2 at 83 GiB, so both suites failed without leaking anything (nightly
run 36659663201). A replicated suite still needs those 90 GiB for its
workers and CDI scratch volumes, so the floor is now checked before the
suite starts. If a node is short there, the suite fails with the
per-node table and an error that blames the sandbox, not the suite.
Cleanup in both lanes now compares against a baseline recorded before
the suite creates anything, and that baseline is no longer free bytes.
On a node whose pools are all thick ZFS, `linstor sp l` does not ask the
satellite for free space. The controller fetches it live only for nodes
with thin pools and otherwise returns the value cached at the
satellite's last report. In LINSTOR v1.33.3 that is
`FreeCapacityFetcherProto.assembleRequests` filtering on `hasThinPools`,
with `CtrlStorPoolListApiCallHandler` falling back to
`getFreeCapacityLastUpdated`. The last report can be taken while `zfs
destroy` is still freeing blocks in the background. A cleanup that had
removed everything then kept polling a frozen figure 0.4 to 0.9 GiB
short and failed after 300s. A byte tolerance does not fix that, because
the residue is already close to the smallest volume a suite is known to
have leaked (1 GiB).
The baseline is now the set of ZFS datasets under each node's LINSTOR
pool. It is read with `zfs list` inside that node's `linstor-satellite`
container. The zpool name comes from the pool's
`StorDriver/StorPoolName` property, because it is `data` on QEMU nodes
and `data-srvN` in container mode. Cleanup waits until no dataset
outside the baseline remains. A destroyed dataset leaves the namespace
when `zfs destroy` returns, whether or not its blocks are still being
freed, so background freeing no longer fails the check. A real leak,
like the 21 GiB scratch volume left on srv3 in an earlier run, is
reported by node and dataset name instead of a KiB difference. A node
whose pool cannot be read fails the check. A run that fails before it
records a baseline no longer compares against the previous suite's file.
Every read the new probe makes runs under `timeout -k 5 30`, so a hung
controller or satellite fails one poll and the wait still ends with its
failure message.
The kafka suite deleted its apps but never their PVCs. The chart keeps
them on purpose with `deleteClaim: false`. One run left 35 GiB in
`tenant-test`, 20 GiB of it on srv2, and the backup round trip left its
two clusters' volumes in `tenant-root` the same way. Each test now
deletes its clusters' PVCs by `strimzi.io/cluster` after the apps are
gone and waits for them. The `kafka` test does it from a step cleanup,
so it also runs on a failed run.
The tests cover both fixes. `hack/linstor-storage-mode_test.bats` has
cases for a clean cleanup, a volume still listed on the first poll and
gone on the next while the cached free capacity stays short, a leaked
volume, a node that cannot be read, a hung satellite read, a missing or
stale baseline, and the replicated-lane floor. Its fake runs the real
controller-side `jq` filter against a LINSTOR-shaped listing.
`hack/kafka-suite-pvc-cleanup_test.bats` checks that each cluster's PVC
delete is in the suite and runs after the app cleanup. One stub line
goes into `hack/run-kubernetes-talos-diagnostics_test.bats`, because
that test calls `cozy_cleanup` with no baseline and only checks the
order of the deletes. The new guards were mutation-tested.
Some limits stay as they are. The local lane waits for dataset names to
disappear, not for the freed space, and has no free-space floor, so a
background destroy can still be freeing blocks when the next suite
starts. The 90 GiB precondition in the replicated lane still reads the
cached free capacity, and its comment says so. The two `kubectl exec`
reads in `cozy_wait_linstor_pool_free` were there before this PR and
still have no time limit of their own.
Two related problems stay out of this PR. The `kafka-metadata` suite
leaks PVCs the same way, tracked in #4641. The stale comment in
`examples/backups/kafka/cleanup.sh` and the missing PVC pre-clean in the
round trip are in #4649.
Closes#4617, closes#4616, closes#4653
### Screenshots
Not a UI change.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(e2e): the kubernetes suites' LINSTOR cleanup compares each node's ZFS datasets against a pre-suite baseline instead of cached free capacity, the replicated lane checks its 90 GiB floor before the suite starts, and the kafka suite deletes the PVCs its clusters keep.
```
Every existing case installs SeaweedFS at one limit and stays there, so
nothing walks the transition an operator makes when lowering it: each
volume at or above the new value leaves the writable set and the bucket
writes again only after a grow. The failure is a PUT that times out
while every release stays Ready, so the suite ends on a write into a
bucket filled between the two limits.
The suite runs its own instance, created directly so the limit can move
without racing the tenant module, and asserts on the live master that
the value set on the resource arrives with the growth counts intact.
seaweedfs changes now select the suite, which the selector's mapping
and its tests are updated for.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Each SeaweedFS chart has its own suite and neither feeds the other, so a
key the app chart writes and the data-plane chart stops reading, or one a
chart bump moves, leaves both suites green. Render the hand-off the way
helm-controller performs it and assert the master's volume size limit
and growth counts on the result.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fixture left TLS unset with external off, so the chart rendered
no CA and no TenantProjection, and no e2e run walked the chain from
the sentinel through the CA-extraction controller to the
tenantsecrets read path. Turn TLS on explicitly, without allocating
a LoadBalancer, and check both ends: the projection is key-free,
carries the chart's CA and is served through tenantsecrets while
the key-bearing Secrets are not; and a client trusting only that
anchor verifies the listener, while a wrong anchor, no anchor and
plaintext HTTP all fail.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
BucketAccess turns accessGranted as soon as the COSI driver has written
the new key into the filer's IAM file. The S3 gateway loads that file
later, from a filer metadata subscription, and with two filer replicas
the change can reach the filer a gateway follows only after the other
replica passes it on. Both bucket suites call mc alias set right after
the assert, so a key the gateway has not loaded yet is rejected with
InvalidAccessKeyId and the suite goes red.
Probe the key through the same port-forward the writes use. Only the
InvalidAccessKeyId text is waited on, for at most 60s; any other error
or a timed-out attempt fails at once, and the window expiring fails
with a message naming the BucketAccess. The two steps get an 8m op
ceiling so the inner budgets still sit under it.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The shared image-ref enumeration had no rule for the upstream
ingress-nginx values layout, where the repository sits in `image` and
the digest in its own `digest` key with no `@`. Both ingress-nginx
refs in the tree (the controller and the default backend) were
therefore invisible to the promote retag, the nightly mirror, the
multi-arch gate and the promoted-packages verifier.
The new rule requires a complete sha256 digest, so upstream's unpinned
default (`digest: ""`) and digest-less image maps emit nothing.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The test that a call cut by its own bound just past the deadline is not
blamed on the budget ran a 3s bound inside a 4s pod section. A
whole-second tick of date +%s between DP_T0 and the first call left the
bound reaching the deadline, the collector then blamed the budget, as
it should, and the test failed. The pod section is now 10s against the
same 3s bound, so a tick cannot reach the deadline.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Ready-conditions and events reads share one block and added a line
of their own when they failed, on top of what their stderr put into
the block. Once dp_run marked every cut and refusal there, a cut read
carried two lines for one event, and before the budget status existed
the second named a timeout the first said the deadline had chosen.
The two reads now send stderr to the scratch file and, after their
output, write kubectl's stderr into the block whole, without dp_run's
marker, plus one line naming the read when it failed. kubectl's
message is kept as it came, since an error can follow lines of
throttling notices on a degraded apiserver. Without a scratch file
their stderr goes into the block directly. The probe comments also say
what the "spent" outcome now covers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Once dp_run marked a deadline cut inside an exec block, the reads that
report their own outcome still read the 124 their shortened bound
produced and called it the read's own timeout, quoting a bound the
deadline had chosen. A Ready-conditions block then gave two reasons
for one cut, and every list and lookup note did the same beside the
section line that blamed the budget.
dp_run now returns a status of its own for a deadline cut, which the
read notes, the node and interface lookups and the LB probe all report
as the budget. pod_on_node takes it from dp_run instead of deciding it
a second time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
An exec block keeps whatever it printed before its bound fired, and
nothing in the block said the bound had fired. The service map every
affected node gets is the largest such block, and a
truncated one answers the question it is there for, whether a backend
is still in the map, wrongly while looking complete.
dp_run now marks a cutoff where the call's output goes, as it already
does a refusal: the deadline's own cut names the budget, any other
names the call's bound. The section note also stops asserting that a
call was cut when the budget only refused the next one.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
dp_run credited a cutoff to the budget whenever the deadline had passed
by the time the call returned. A call under its own, unshortened bound
can fire, or come back from its kill grace, just past a deadline it
never reached; the section note then said the budget cut a call, next
to the read's own note naming its own timeout, and the walk closed as
cut short by the budget. The bound alone is not enough either: a call
killed early, as an OOM-killed remote process is, returns 137 under a
bound cut to the deadline long before the deadline arrives.
dp_run and pod_on_node now share one rule: a cutoff is the budget's
only when the call's bound reached the deadline and the deadline has
passed, and never when no timeout binary bounded the call at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A walk the budget cut short closed on "see the section line above",
but that line goes only to capture-notes.txt, while the closing line
also reaches the job log, where nothing sits above it to see. It now
names the file.
The budget knob replacement and the run without a timeout binary had no
test of their own; both are now pinned.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The LoadBalancer walk can stop on the budget, as the pod walk can, and
still closed on "capture complete" right under the line naming what it
did not probe. Both walks also called themselves complete when they
ended on their own after the deadline had cut a call inside their last
item.
Each walk now closes as stopped early when it broke out of the loop,
and as cut short by the budget when this run's section note says a
call was cut or refused; the note's own text is the evidence, since the
clock read after the loop can pass the deadline without any call having
been cut. The LB cap break, which shares that closing line, closes the
same way.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pod walk can now also stop when the pod section's share of the
budget runs out, as it already could at the pod cap. Both exits name
what they left behind, and the line after them still announced the
capture as complete. It now says the walk stopped early.
The helper comment about the run separator also named every capture
file, while the OVN dump and the speaker logs are still replaced on
each run; it now lists the files it covers.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Once bounds are clipped to the budget, a cutoff has two meanings: the
call's own bound fired, or the deadline shortened it. Node lookups cut
the second way were still written up as lookups that did not answer, a
cause about the cluster that nothing observed. And the section note was
written only when a call was refused, so when the deadline cut a
section's last call the truncated capture carried no word about the
budget at all.
A lookup cut under a bound the deadline shortened now names the budget
in the node capture, the endpoint interface lookup and the LB probe,
decided from the bound it ran under rather than from the clock when its
status is read. A call the deadline cuts also writes the section
note, as a refusal does. The heavy-capture line also stops claiming
every probe failed when some of them were stopped by the budget.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A few reads that go through dp_run still failed or fell back without a
trace. A speaker log read that failed looked like a
speaker that never announced the address; it now writes a note. The
endpoint tcpdump's fallback to the geneve interface is stated in the LB
capture. The node capture now carries the run separator the pod and LB
files use, through one helper.
Two budget interactions named a cause that never happened. A per-read
timeout knob of zero made every read look refused by the budget, and a
value timeout cannot parse failed every read with 125, reported as
kubectl failing; both knobs are now validated like the budget ones. An
LB probe whose cni-server lookup the deadline cut was reported as a
second, unanswered lookup; it now counts as stopped by the budget.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The budget can run out inside one capture rather than between them. A
node lookup refused by dp_run was then written up as a lookup that did
not answer, an LB probe whose cni-server lookup was refused as the same,
and a probe exec refused, or cut by a bound the deadline had shortened,
as a node with no cni-server or no probe client. Each names a cause that
never happened.
The lookup's own status now reaches its callers, the node capture says
the lookup was not asked, and the probe reports a "spent" outcome that
the decision weighs like an unrun probe and the reason line attributes
to the section's budget. A reserve at least as large as the budget is
also named, since the pod section then starts no call at all.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The collector's default budget is sized under the 300s Chainsaw
backstop. The cozytest.sh caller allows 600s, and on the default it
would give up more than half of that window, so it passes a budget
matching its own backstop. The budget test reads each caller's
override and fails when a budget leaves more of its backstop unused
than the backstop's kill grace.
The Chainsaw call-site comment still described the fan-out caps as the
bound, and the e2e testing guide described an EndpointSlice exception
to the notes contract that the code does not have; both now describe
what the collector does.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The pod section stops a reserve short of the end of the budget and the
LoadBalancer section keeps starting calls after it, but a refusal was
reported as the whole run's budget running out, which the LB capture
written below it contradicts. The note was also deduplicated once per
run, so the LB section running out later left no line of its own.
Refusals now name their section, and the note is written once per
section. Both budget knobs are also read as decimal: shell arithmetic
took a leading zero for octal, so 08 aborted the run before it wrote
a note and 010 meant 8.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-LB capture and its two tcpdump files joined namespace and
Service name with a hyphen, which both may contain, so a-b/c and a/b-c
wrote the same files, and each run truncated them while the notes
beside them accumulate. They now use the separator and the run marker
the per-pod files use.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-node lookup memo keeps a cutoff so a hung lookup is not paid
for twice. Once bounds are clipped to the budget, a lookup can be cut
after a second or two because the pod section's deadline arrived, and
caching that made the LoadBalancer section report the component as
unknown for the node while its reserve still had time to ask.
A cutoff is now cached only when the lookup ran under its full bound.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
A call dp_run refuses for a spent budget returned silently, and every
exec block swallows its status, so a capture the budget ran out in the
middle of kept only section headers. A header with nothing under it
reads as a command that found nothing, and the notes said nothing
either, because the loops only decline whole pods and LBs.
The refusal now goes to stderr, which lands in the block it emptied,
and capture-notes.txt gets one line per run. The OVS interface lookup
also stops calling a failed, cut or refused lookup an absent interface.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The fan-out caps counted pods and LoadBalancers while both callers'
backstops count seconds, and a single affected pod's worst case already
ran past the 300s Chainsaw one. No count could be made to fit, and a
cut-off run lost whatever came last, which was always the LoadBalancer
capture.
Every kubectl call now goes through dp_run, which clips the call's
bound to what is left of a run-wide budget sized under the smaller
backstop and starts nothing once it is spent. The pod section stops a
reserve short of the end so the LoadBalancer section always gets time
to start. A read cut by a clipped bound quotes the bound it ran under,
a read never started says so instead of naming a timeout, and a section
that runs out of budget names what it did not capture.
Tests pin the budget against both callers' backstops, the behaviour on
a cluster where every read hangs, and that no kubectl call bypasses
dp_run, listing any that does by line.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three collector tests shrank both per-read bounds to one second to make
the read under test fail fast. The same bounds also cover the reads that
must answer, so on a loaded machine the service, endpointslice or pod
list is cut instead, the collector correctly takes another branch, and
the test fails as if the code under test had regressed.
A `timeout` shim on PATH now cuts only the reads whose argv the test
names and runs the rest unbounded against the instant stub, so the cut
is the input rather than a race. A cut read that bypasses `timeout`
still reaches the stub's sleep and runs into the test's own backstop. The
tests also assert the collector finished on its own and the decision
line it logs, not only that a file exists.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
`cilium-dbg bpf lb list` ran only inside the LoadBalancer path, which is
entered only when a LoadBalancer IP fails its probe. The map describes
every service translation on the node, so a pod that cannot reach a
ClusterIP left no record of which backend the VIP resolved to or
whether a backend that stopped answering was still in the map.
It now runs unfiltered in the per-node capture, which runs for the node
of each affected pod the collector reaches.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
POSIX sh has no local scope, so every assignment in a function lands in
the caller. capture_node wrote `node`, the variable of the loop that
calls it, and capture_lb_node wrote the LB loop's _lbip, _np, _epip and
_of. Both stayed harmless only because each caller passed what it
already held; any other argument would have made the rest of the
iteration read another node's pods and file them under the right name.
Both functions now use their own prefix, like the newer helpers in the
file, and a static test fails when a function assigns a name that a
`read -r` loop or another scope also owns.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The per-pod capture file joined namespace and pod name with a hyphen,
which both may contain, so a-b/c and a/b-c wrote the same file and the
second capture silently replaced the first. An underscore can appear in
neither, so it keeps every pod apart.
The file was also truncated on each run while capture-notes.txt beside
it accumulates, leaving notes from two runs next to evidence from one.
It now appends with the same run separator the notes use.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>