kubectl exec passes the remote command's status through, so a
non-zero exit after output does not say whether the stream broke or
the command finished with an error. zfs list is the common case: it
lists every dataset it can open, skips one it cannot and exits 1, so
a complete listing got a TRUNCATED marker and a note claiming it held
only what was streamed before the exit.
The in-file marker now names the exit status and says the status
cannot separate the two, and the note beside it says the same. The
timeout branch keeps its own wording.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The report carries the LINSTOR resource list and the pool totals, but
nothing that names the datasets holding a pool's space, so a leftover
zvol, a snapshot, a clone and pool metadata cannot be told apart from
the artifact alone.
Each satellite now lists every dataset of the ZFS pools LINSTOR
records for its node, snapshots included, with the pool's pending
frees. Sizes are exact bytes with avail, so the sizes of leftover
datasets can be summed against the pool root's avail; the rounded
form loses a sub-GiB difference on a large pool. That avail can
legitimately differ from `linstor sp l`, which serves a cached free
capacity on nodes with only thick pools. The pool names come from
LINSTOR rather than from a name pattern. One pool that zpool cannot
open does not cost the listing of the others, and a satellite whose
node name cannot be read says so instead of pointing at the pool map.
The controller's volume list is recorded next to the resource list.
Everything runs on every job, so a red listing can be read against a
green one, and goes through the bounded exec reader like the other
LINSTOR reads.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
CDI deletes an importer pod once, right after it marks the PVC import
Succeeded, and never retries that delete. When that one delete
does not land, the Completed pod keeps pvc-protection on the PVC and a
later DataVolume delete hangs. The report had no evidence for why: the
broken-pod walk skips Completed pods and the cdi-deployment log was
never read.
Read the cdi-deployment log (current and previous) and every pod
carrying CDI's app label as YAML, across namespaces and phases. Only
the log reads wait on the cdi-deployment probe: an importer pod is
owned by its PVC, so it can outlive the Deployment, and a missing
controller must not also drop the pod it left behind.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The tenant node-join wait no longer discriminates. Nested
virtualisation on the shared runners degrades far enough that a guest
which registers in minutes on real hardware can miss any deadline this
test can afford, so a red on that one wait says as much about the
machine the run landed on as about the product, and it teaches a
reader to re-run the job.
The suite still fails on it. That is what keeps Chainsaw's catch
running: the previous-instance container logs, the host snapshot, the
data-plane capture and the event dump are all keyed on the test
failing, and the report stays honest. What changes is one layer up.
The wait writes a marker beside its diagnostics, and a new lane-side
gate reads that marker to decide whether the job blocks. It opens only
when every failed test in the run carries one, and blocks on anything
it cannot attribute: a report it could not read, a failed test the
report does not name, a failure with no marker.
Only the lanes that block a merge ask it. The two pull-request lanes
do; nightly and the tag lane run the same suites and stay hard. A soft
pass on the tag lane would ship a release candidate whose tenant
clusters brought up no worker nodes, and nightly is the only place an
e2e failure reaches anyone who did not open a pull request, since no
lane in this tree notifies anywhere.
A softened red finishes the job green, so the job's result stops
answering whether the suite passed, and both kinds of consumer are
handed the decision instead of inferring it. A step gated on the job
failing would silently stop running: the SSH breakpoint is that case,
and it still opens for a pull request marked for debugging. A job that
reports the result would silently start reporting a pass: the required
commit status now says the deadline was missed and that the merge is not
blocked on it, rather than that the suite passed.
The deadline goes to 29m. That is the test's own budget and nothing
more: its clock starts after the machinedeployment wait, the
LoadBalancer assignment and the healthz probe, while the chart's
autoscaler and health-check timers have been running since the Machine
was created, so no ordering against them is claimed. The chart's 20m
and 30m give a scale to pick against, and a budget over the first
leaves room for a remediation cycle without promising that one fits.
The operation ceiling both suites give the script rises with the
deadline, and the diagnostics phase budget rises inside it, because
the collectors gated ahead of the serial console cost more between
them than the old budget covered. Both budget guards read their terms
from the sources that set them. The e2e job cap rises with them, and
its guard holds what a guard can hold: the pair's operation and catch
ceilings, and one figure across every lane that runs the suite. It
also records that the suites' full summed ceilings have never fit any
cap in that file, so the margin above the bound is taken from a
measured run instead. The prose that restates the cap elsewhere moves
with it, and the sweep that holds the deadline's restatements now
reads the figure spelled in words as well as in minutes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
hack/cozyreport.sh never captured CertificateSigningRequests, leaving the
hypothesis that a node join hangs on kubelet certificate issuance
(bootstrap token, CSR, approval) unverifiable from the collected
artifact.
Add a bounded CSR list and a per object yaml capture for the host
cluster, next to the existing node walk, and the same capture for every
tenant Kubernetes cluster whose kubeconfig a failed kubernetes-*
Chainsaw test left on disk. Both walks reuse the existing bounded read
helpers, so a cut off or refused read produces the same truncation and
failure markers every other capture in the file already writes, and an
empty CSR list is not treated as an error.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
An e2e lane asks for a runner class, not for a machine. A flex shape is
filled from whatever capacity the pool has at the time, so one lane can
be served by two processor generations, and the region is a preference
with a fallback. None of that reached the report, so a red run and a
green one could not be compared by the hardware they ran on.
The read is taken on the runner VM, before the sandbox container is
created, because the metadata service answers on a link-local address
the container has no guaranteed route to: an "unreachable" answered from
in there would look exactly like a capture that had tried. It is wired
into the sandbox apply target rather than into a workflow step, so every
lane picks it up through `make prepare-env` on both the passing and the
failing path, and cozyreport folds the file into the artifact from
inside without going near the endpoint.
The keys are fetched one at a time and the root instance document never
is. That document carries the instance and compartment OCIDs, the image
OCID, the display name, the hostname and the user metadata bag, and the
report is a public artifact, so asking per key makes that property
something the wire settles rather than something a filter has to be
trusted for. It also matches where the shape sizing is published: it has
its own subpath and is not in the root document at all. From the sizing
object only the numbers are kept, so the rule holds for keys nobody has
seen yet, and an answer that is not one scalar is named rather than
printed.
Every read carries a wall-clock ceiling, since a read that never returns
costs the step its whole budget without reaching any exit status, and the
first key doubles as the probe so a machine that is not an OCI instance
pays one ceiling rather than one per key. Nothing here is fatal: this
runs inside the one retried step in the job, so a runner with no
metadata service leaves an "imds unreachable" line and an exit status
of 0.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and
the rest -- nodes, namespaces, events, HelmReleases, packages,
packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads
and every module gate -- read with none. The tarball is written on the
collector's last line, so a single read that never returns does not
truncate the report, it loses it, on the run where it is the only
evidence left.
Each read now goes through the shape that fits it: the existing bounded
reader for a read whose output is a file, the existing selector for one
that feeds a loop, and a new probe for the `if kubectl get deploy ...`
gates. A gate is a read like any other, and an unbounded one is worse
than the reads it guards, since the module below it never starts. Its
cut-off is recorded, because skipping the module leaves the same empty
tree as a cluster that never had the component; a gate that answers "not
installed" is recorded nowhere, since that is the cluster answering
correctly and a marker firing on every install without KubeVirt is one a
reader learns to skip. The reads that stay on the bare wrapper say so
where they sit; what they share is that the output is not the reader's
to write, going into a variable or through a pipeline that transforms
it, so there is no moment at which a marker could attach.
The header claimed a per-section list of what was bounded, and the tests
froze that list plus a per-section count of the unbounded reads. Both are
gone. The list had already gone stale, naming four sections while the
file had ten, and the counts were an undercount by construction -- they
matched a redirect to `$VAR/`, which no gate, selector or command
substitution has. One test replaces both: no read below the library guard
runs without a ceiling, checked by looking for a counterexample.
A ceiling per read is not a ceiling on the report, and the header now
says so where it used to imply the opposite. The pod, VM, VMI and
services walks carry a count cap and a time budget; the walks over nodes,
namespaces, HelmReleases, packages, PVCs and certificates carry neither,
so their cost is the cluster's to set. What this removes is the read that
never returns.
The bounded reader now names the tool it ran and describes what that tool
returned. It served kubectl alone until this change gave it talosctl and
the four host commands, so a `df` that hit a stale mount left a marker
reading "kubectl exited 1 ... not because the object ended" -- two false
statements in one line, in an artifact whose premise is that its notes
can be believed.
The exec reader keeps the remote command's whole message in the listing
it belongs to. `kubectl exec` appends its own `command terminated with
exit code N` after whatever the command said, so quoting the last line
there drops the cause and keeps the trailer, and sending the message
away from the file leaves a zero-byte listing -- the "returned nothing,
or never ran" question this collector exists to answer. Its cut-off note
also stops asserting that the archive is beside it: a stream that got
members across is kept and truncated, one killed before its first byte
leaves zero bytes and the call site's emptiness check removes it, and
the reader runs before that check, so it names both.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The LINSTOR module runs six reads through `kubectl exec`: three
listings, the error-report index, and the controller and per-satellite
ErrorReport bundles. They are the only reads in the collector whose
duration is set by a workload rather than by the apiserver -- a wedged
controller, a satellite that stops answering, or a broken SPDY upgrade
holds the stream open with no clock anywhere in the path. The tarball is
written on the collector's last line, so one of those never returning
does not truncate the report, it loses it.
They go through a reader of their own rather than the existing one.
`kubectl exec` returns the remote command's exit status, so a 124 there
is either this collector's deadline expiring or a command that itself
exited 124, and the status does not separate them; describing it the way
every `get`, `logs` and `describe` is described would state a mechanism
about the machine writing the report for something that may have
happened inside a pod. The note says what is open instead of resolving
it, and goes beside the output rather than into it, since two of the six
stream a gzipped tarball that a line of prose turns into an archive
nothing can open. kubectl's own message travels with the note for the
same reason -- until now it was merged into the listings, so every run
put the "Defaulted container" line at the top of a table.
The guard that held the old assumption asserted only that no bounded
read was an exec, which leaving every exec unbounded satisfies. It now
also requires each exec to be bounded and requires the reader not to
borrow the classification written for reads whose 124 is unambiguous.
Matching on the verb rather than on the string `kubectl exec` is part of
the fix: half of these calls spell it `kubectl -n cozy-linstor exec`,
which that string never saw.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The step that runs hack/cozyreport.sh carried no timeout-minutes in any
of the three workflows that run it, so it inherited only the job's 180.
The collector writes its tarball on its last line and the upload is the
next step, so a read that never returns spends what is left of the job
budget, the job is cancelled, and the upload never runs -- the artifact
is lost entirely rather than truncated, on exactly the run where it is
the only evidence left. `|| true` guards against a non-zero exit, not
against a read that does not return.
30 minutes is measured against the step rather than divided out of the
job: it takes 3-42s across sampled runs, on failed e2e jobs as well as
green ones, and 30 leaves five sixths of the job budget for the steps
that upload what this one produced.
A test pins the ceiling in all three workflows and requires it to stay
under the job's own, since a step ceiling at or above the job's cap is
the same as no ceiling at all.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The failure snapshot selected pods by their STATUS column, so a pod
whose container runs while its readiness probe never passes -- STATUS
Running, READY 0/1 -- was dropped from the report entirely. That is a
common way for a suite to time out: the replica never joins, its owning
HelmRelease sits in InProgress, and the artifact carries no trace of the
pod the run died on. Select on the READY column instead, which also
covers the partially ready multi-container case, and keep every phase
that was collected before. On one degraded cluster that is four of the
seven broken pods the old filter reported nothing about. The VMI and VM
sections had the same blind spot and now share one selector that reads
READY as the last field, because a Pending object prints fewer columns
than a Running one and the two kinds differ only in how many columns
precede READY. READY counts ready containers rather than the pod's Ready
condition, so a pod held back by an unsatisfied readiness gate is out of
scope; a test pins that rather than leaving it to be found in an empty
directory.
Logs are requested for every container of a selected pod rather than
skipped for Pending ones, and the per-pod collection moved into a helper
so it can be driven against a stub kubectl. kubectl's refusal is the
finding, and an empty result is ambiguous on its own: a read against an
unscheduled pod exits 0 having printed nothing, which is what a
container that started and stayed quiet also leaves. Every site that
classifies a failed read now orders its branches the same way, naming
the timeout first where both a cutoff and an uncapturable stderr apply.
The CRD gate in the summary had no such branch at all: it returned in
silence, which is byte-identical to the CRD being absent, so every
section behind that gate vanished from a report whose only real problem
was that the machine writing it had nowhere to put a temp file.
Collection used to stop silently. A walk that hit its cap said so in the
job log and nowhere in the artifact. Now the pod walk leaves
COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or
COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off
part way through, COLLECTION-FAILED.txt when the pod list never came
back rather than the empty tree a healthy cluster produces, and
COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a
fact about the machine writing the report and not about the cluster.
Tenant namespaces are collected first, so a cap that only bites on a
broadly degraded cluster is not spent on platform pods while the pod the
suite died on falls off the end.
Container logs are read with --tail=2000 where they were previously read
whole, and every bounded file states the bound it was written under,
because --tail drops the oldest lines and nothing else in the output
would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log.
The bounded whole-object read no longer merges stderr into pod.yaml,
vm.yaml and vmi.yaml, which are read by yq rather than by eye: a
warning on an otherwise complete read left an unparseable file. stderr
goes to a scratch file and is copied in only when kubectl returned no
object, and the truncation markers those files carry are written as YAML
comments, since a bare marker line after a complete mapping is a second
top-level node.
Six latent faults surfaced while making these reads report. The kamaji
module was gated on the LINSTOR controller, so a cluster without LINSTOR
got no kamaji evidence at all, which is what a tenant-kubernetes
bootstrap timeout needs most. The Flux sources section nested one
jsonpath filter inside another, which kubectl rejects outright, so it
printed nothing on every report ever taken and read as "all sources
Ready". The HelmRelease listing took a single field as STATUS where
STATUS is the Ready condition's whole message, so every failed release
rendered as one word. The services section filtered for a pending
LoadBalancer address on the CLUSTER-IP column, which never carries that
value. The cozystack module read its image from a Deployment the
installer stopped shipping when it moved to the operator, putting
kubectl's NotFound in the first file of the report. And the cozystack
apps section gated all three of its kinds on a CRD in an aggregated API
group, so it never ran once; two of those names were wrong, and both
are collected again.
The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats
test that installs one and then fails prints no TAP line at all, so
output grepped for `not ok` reads as green, while hack/cozytest.sh
reports the same failure correctly. A guard enforces that over a named
list of files rather than a glob with exceptions, because a list of
exceptions is the same claim written so that forgetting to shorten it
makes it false.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The linstor-Satellite pod stdout log references its error reports
only by ID:
[DeviceManager] ERROR LINSTOR/Satellite/... SYSTEM - Failed to
create zfsvolume [Report number 6A4A394E-E4EA0-000000]
The body of that report (which carries the actual `zfs create -V …`
stderr, the DRBD adjust output, the storage-driver stack trace, and
the exit code) lives at `/var/log/linstor-satellite/ErrorReport-*.log`
on the satellite pod. crust-gather only captures pod stdout, so the
report body stays on the pod's ephemeral filesystem and is lost when
the e2e sandbox is torn down at end-of-job.
Consequence for post-mortem: a run that failed on a
`ResourceExhausted` / `Not enough available nodes` CreateVolume retry
loop shows only the driver-level symptom in cozyreport. Root cause
attribution (out-of-space, dataset name conflict, ZFS module error,
kernel signal) is not distinguishable without the report body, and
without the sandbox is unrecoverable.
Extend the linstor module in `hack/cozyreport.sh` to tar the
`ErrorReport-*.log` set off every `linstor-satellite` pod into
`linstor/error-reports/<node>.tgz`. Guarded so a satellite with zero
reports (the healthy case) leaves nothing in the tree, and a
transient exec / tar failure never fails the surrounding cozyreport
run.
Verified against a live dev cluster: label selector
`app.kubernetes.io/component=linstor-satellite` matches all three
satellite pods, and the tar bundle for one active satellite comes
out at ~60 KiB. No product-side change.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
cozytest.sh runs each .bats as one invocation, exits on the first failing @test, and never invokes bats setup/teardown. Add a single EXIT-trap hook: on failure, snapshot the host AND every nested tenant cluster (tenantkubeconfig-*) with crust-gather into per-app dirs BEFORE cleanup — so each failure is independently inspectable (crust-gather serve) instead of one end-of-suite host-only dump; then ALWAYS run the file's cozy_cleanup() if defined.
run-kubernetes.sh gains cozy_cleanup() (delete tenant clusters, wait for PVCs to free): a failed kubernetes test otherwise leaves tens of GiB of worker-VM PVCs in tenant-test, exhausting the shared tenant-quota and cascade-failing every storage-heavy app after it. The kubernetes .bats now source run-kubernetes.sh at file scope so the parent-shell trap can reach the hook.
Ship pinned crust-gather 0.15.2 in the e2e-sandbox image; fold per-test snapshots into cozyreport.tgz.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Run 27057500380 failed on harbor's BucketClaim never reaching
bucketReady=true, and neither artifact could explain why: the per-app
diagnostics dump has no bucket objects, and cozyreport collects
neither COSI resources nor objectstorage-controller / seaweedfs
provisioner logs — by the time the report is taken, the bucket state
has moved on.
Add a COSI module to cozyreport.sh (bucketclaims/bucketaccesses/
buckets/classes as custom-column tables plus full YAML, controller and
per-namespace provisioner logs) and a bucket-readiness snapshot to the
per-app failure dump in the PR workflow. The COSI CRDs ship no printer
columns, so plain kubectl get would show only NAME/AGE — readiness
fields are pulled explicitly.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- Resolve hack/cozyreport-summary.sh via SCRIPT_DIR so cozyreport.sh
can be invoked from any working directory, not only the repo root.
- Discover Talos node IPs via `kubectl get nodes` (with the existing
192.168.123.11-13 list as fallback) instead of hardcoding the E2E
sandbox topology.
- Replace broken `awk '$NF != "True"'` filter on the cozystack-apps
loop with a jsonpath check against the Ready condition; the previous
filter compared AGE ("10m" etc.) against "True" and let every
resource through, defeating the non-Ready filter.
- Switch the Flux source CRD summary loop from column parsing to
jsonpath (column positions differ between HelmRepository and
OCIRepository printer columns) and add ExternalArtifact, matching
what cozyreport.sh already collects.
- Add `kubectl get crd` guards before HR/source CRD queries so the
summary script does not abort under `set -eu` when Flux v2 is not
installed.
- Relabel "OOMKilled in last 30 min" to "Recent OOMKilled events
(last 20)" — the kubectl call only limits count, not time.
Surfaced from coderabbit and gemini review on #2553.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`hack/cozyreport.sh` now collects materially more signal so a triage
artifact can show what is broken rather than just dumping pod logs:
- Flux controller logs (helm-controller, source-controller,
notification-controller, kustomize-controller, last 2000 lines)
- Flux source resources (HelmRepository, OCIRepository, GitRepository,
ExternalArtifact)
- Decoded Helm storage secrets for non-Ready HRs
(`base64 -d | base64 -d | gzip -d`)
- Cluster events (all + warning-only filter)
- cert-manager Certificate / CertificateRequest / Order / Challenge
resources + cert-manager logs
- `cozystack-operator` deployment logs (current + previous)
- `Application` / `ApplicationDefinition` / `Tenant` resources
- Sandbox host context per node: df, free, ps, dmesg,
talosctl logs/dmesg/kubelet/containerd
New `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive
root listing what is broken right now — first thing to read when
triaging an artifact.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
## What this PR does
Previously the debug log collection script that fired when CI failed
treated Packages and PackageSources as namespaced resources and as a
result of incorrect parsing failed to correctly kubectl describe and
kubectl get -oyaml them. Additionally, the script did not read the logs
of init containers. These issues are fixed with this patch.
### Release note
```release-note
[ci] Improvements to cozyreport.sh (ci log collection script): fix
retrieval of Package and PackageSource details, consider initContainers
as well as containers, when fetching logs of errored pods.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Some "while read NAMESPACE NAME _" steps in the cozyreport script that
collects debug info weren't omitting the headers in `kubectl get` output
and trying to get objects named NAME in namespace NAMESPACE. This patch
adds `--no-header` to some places where it was forgotten.
```release-note
[ci] Fix an error in cozyreport that tried to parse non-existent objects
and generated garbage output in CI debug logs
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Aleksei Sviridkin