mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-04 02:57:27 +09:00
docs/system-memory-limits
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 myasnikovdaniil
|
2db18a0c86 |
test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport (#4657)
## What this PR does The kubernetes suites' LINSTOR cleanup check now compares dataset names per node, and prints the leftover names when it fails (#4651). This PR puts the rest of what a triager needs into `cozyreport.tgz` on every run, green or red: the size of each dataset, its `origin` and `creation`, the snapshots, the pool's pending frees, and the LINSTOR volume list. What `hack/cozyreport.sh` now collects: - `linstor/zfs-pools.txt`: the zpool behind each ZFS storage pool, per node, taken from `StorDriver/StorPoolName` in `linstor --machine-readable sp l`. - `linstor/zfs/<node>.txt`: that node's satellite runs `zpool get -p freeing,leaked,allocated,free` and `zfs list -p -t all -o name,used,avail,refer,origin,creation -r` over the node's pools. - `linstor/volumes.txt`: `linstor v l`, next to the existing `resources.txt`. Sizes are exact bytes, so the sizes of leftover datasets can be summed against the pool root's `avail`. That `avail` can differ from `storage-pools.txt` without anything leaking: on a node with only thick pools, `linstor sp l` serves the free capacity cached at the satellite's last report. A dataset LINSTOR could not destroy because snapshots depend on it is renamed to `_deleted_<name>` in the same pool, so a search for leftovers has to look for that prefix as well as `pvc-`. A pool that `zpool get` or `zfs list` cannot open does not cost the listing of the other pools. `zpool get`'s message and exit status go into the listing, and `zfs list` lists the rest. A node LINSTOR names no ZFS pool for gets a note instead of a listing, because `zfs list -r` with no pool argument lists every pool on the machine. A satellite whose `spec.nodeName` cannot be read gets a note that says so. All reads go through the bounded exec reader, like the other LINSTOR reads, and the time budget in the script header counts them. The second commit changes how that reader marks a read that wrote output and then exited non-zero without hitting the timeout. It used to append `TRUNCATED ... part way through, not because the command finished`. But `kubectl exec` passes the remote command's status through, and `zfs list` over one missing pool prints a complete listing of the others and exits 1. The marker now names the exit status and says the status can't tell an error after complete output from a cut stream. This changes the marker for every LINSTOR exec read. The timeout marker stays as it was. Tested with `hack/cozytest.sh hack/cozyreport.bats`. The new tests run the controller and satellite shells against stub `linstor`, `zpool` and `zfs`, with the real `jq`. They cover a failed and an unparsable storage-pool read, a pool row with a missing, empty or blank backing name, a non-ZFS pool, a pool name with a space, a pool that `zpool` and `zfs` cannot open, a failing `zfs`, and an unreadable node name. A test goes red if any of these break: the filter that keeps non-ZFS pools out, the filter that drops a missing, empty or blank pool name, the jq exit status, the `zpool` fallthrough, the node-name note, or the reader's new wording. Refs #4653 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): cozyreport records the ZFS datasets of each node's LINSTOR pools in exact bytes, and the LINSTOR volume list, on every e2e run. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Collection reports now include LINSTOR volumes, ZFS pool capacity and usage, and per-node ZFS datasets and snapshots with exact-byte details. Dataset listings are scoped to the ZFS pools LINSTOR reports for each node. * Reports include notes or warnings when node, pool, or dataset information is missing or cannot be collected. * **Documentation** * Clarified storage report contents, artifact markers, bounded reads, and how unsupported data, incomplete results, collection limits, and per-object warnings are reported. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2e5fd56954 |
fix(e2e): stop calling a failed exec read after output a cut
kubectl exec passes the remote command's status through, so a non-zero exit after output does not say whether the stream broke or the command finished with an error. zfs list is the common case: it lists every dataset it can open, skips one it cannot and exits 1, so a complete listing got a TRUNCATED marker and a note claiming it held only what was streamed before the exit. The in-file marker now names the exit status and says the status cannot separate the two, and the note beside it says the same. The timeout branch keeps its own wording. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d195a56793 |
test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport
The report carries the LINSTOR resource list and the pool totals, but nothing that names the datasets holding a pool's space, so a leftover zvol, a snapshot, a clone and pool metadata cannot be told apart from the artifact alone. Each satellite now lists every dataset of the ZFS pools LINSTOR records for its node, snapshots included, with the pool's pending frees. Sizes are exact bytes with avail, so the sizes of leftover datasets can be summed against the pool root's avail; the rounded form loses a sub-GiB difference on a large pool. That avail can legitimately differ from `linstor sp l`, which serves a cached free capacity on nodes with only thick pools. The pool names come from LINSTOR rather than from a name pattern. One pool that zpool cannot open does not cost the listing of the others, and a satellite whose node name cannot be read says so instead of pointing at the pool map. The controller's volume list is recorded next to the resource list. Everything runs on every job, so a red listing can be read against a green one, and goes through the bounded exec reader like the other LINSTOR reads. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov Daniil
|
cc942fed68 |
test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting. The unit lane needs complete TAP results and JUnit reports without changing how live-cluster suites run. Sourced production helpers still need a dash pass, and unit test bodies need nounset without exporting shell options into subprocesses. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
cd448fdc04 |
test(bats): widen the comments describing the errexit exemption
Six suites explain why they avoid `! cmd` by saying errexit is suppressed for a "negated pipeline". The exemption is not about pipelines: a bare `! true`, a negated `[ ... ]`, one inside a `for` body or a brace group, one in a case branch, and one following `;`, `&`, `&&` or `||` are all exempt too. Anyone reading the narrower wording could conclude the other shapes are safe. One of the six carries the sentence twice, and the second copy is split across two physical lines -- `` `!` `` ending one and `pipeline)` opening the next -- so a line-oriented search for the phrase does not see it. Prose wraps; a grep for it has to fold the lines first. Comment text only; every one of these files already uses the working form the comment recommends. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d918887aac |
fix(e2e): collect the CDI controller log and CDI pods in cozyreport
CDI deletes an importer pod once, right after it marks the PVC import Succeeded, and never retries that delete. When that one delete does not land, the Completed pod keeps pvc-protection on the PVC and a later DataVolume delete hangs. The report had no evidence for why: the broken-pod walk skips Completed pods and the cdi-deployment log was never read. Read the cdi-deployment log (current and previous) and every pod carrying CDI's app label as YAML, across namespaces and phases. Only the log reads wait on the cdi-deployment probe: an importer pod is owned by its PVC, so it can outlive the Deployment, and a missing controller must not also drop the pod it left behind. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
e6be333c06 |
test(e2e): capture CertificateSigningRequests in cozyreport
hack/cozyreport.sh never captured CertificateSigningRequests, leaving the hypothesis that a node join hangs on kubelet certificate issuance (bootstrap token, CSR, approval) unverifiable from the collected artifact. Add a bounded CSR list and a per object yaml capture for the host cluster, next to the existing node walk, and the same capture for every tenant Kubernetes cluster whose kubeconfig a failed kubernetes-* Chainsaw test left on disk. Both walks reuse the existing bounded read helpers, so a cut off or refused read produces the same truncation and failure markers every other capture in the file already writes, and an empty CSR list is not treated as an error. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
575b41e1cd |
test(hack): let each bats file declare the EXIT traps it still holds
The ban on EXIT-trap cleanup in hack/*.bats was enforced by two lists inside hack/cozyreport.bats: one naming the files known to be clean, one freezing an exact trap count for each file that was not. Both had to be edited from whatever change moved them, and that is where the guard kept failing. A change adding a trap to its own file had to edit a string in a suite it otherwise never touches, so two changes sharing no line still invalidated each other: each stayed green against its own base, git merged both cleanly, and the guard went red only once the second landed. A file arriving with traps hit the same wall from the other side -- the inventory did notice it, since the string it compared was built by scanning the directory, but absorbing it meant an edit in a file nothing in the author's diff pointed at. Replace both lists with a declaration each file makes about itself: one "# EXIT-TRAP DEBT: N" comment, exact rather than a ceiling. A file carrying none must install no EXIT trap, which covers the converted files, the files that never had one, and every file added later. Growing or shedding a trap now fails in the file the change already edits, so two changes that disagree about a count collide textually instead of silently. The declaration is read from the leading comment block, not from anywhere in the file. A .bats file is shell that writes shell, so the same line turns up inside a heredoc, a fixture writer or an expected-output string, where it is data belonging to one test. Reading it there as a statement about the whole file would let an unrelated fixture excuse a real trap, and would do it silently, since nothing in that test's own diff looks like a declaration. A comment block is the region with no interior: stopping instead at the first @test would still read a line out of a helper's heredoc. Not every counted handler is debt. A trap inside an explicit subshell does not replace the bats binary's own, so a test failing inside `( ... )` still prints its `not ok`. hack/e2e-test-openapi.bats kills a backgrounded kubectl proxy that way, and moving the kill to the end of the body would leak a process holding a fixed port. Its declaration records that rather than scheduling a conversion, and because the ratchet is exact, removing the trap fails too -- the count protects the construct. What the count cannot do is tell the two apart: substituting a test-level trap for the subshell one keeps the total at 1 and stays green, which the header states rather than leaves to be found. Counting bounds the keyword and the signal the same way, at any character that cannot be part of an identifier, and matches the signal in either case. Whitespace on the right missed `trap ... EXIT; cd "$tmp"`; whitespace on the left missed `tmp=$(mktemp -d);trap ... EXIT` and `(trap ... EXIT; true)`; upper case missed `trap ... exit`, which bash and dash both install. All are real handlers that scored zero, and the left boundary matters most, since the inventory being replaced had none and caught the semicolon form. A bare word boundary is not enough either way: `bootstrap ` ends in `trap `, and it must keep scoring nothing. A quoted signal counts for the same reason as the rest. Two handlers sharing one line are reported rather than counted, since the count is a count of lines and the second would otherwise arrive without moving the total; splitting the line properly needs a shell parser, a semicolon inside a handler's own action not being a separator, and guessing wrong undercounts -- the one direction a ratchet cannot afford. The scan recurses, so hack/e2e-apps/*.bats is covered rather than sitting one directory below the guard that claims the tree. It reads .bats and nothing else, so a handler arriving through a sourced .sh stays outside it -- hack/e2e-chainsaw/_lib/run-kubernetes.sh installs two, each benign for its own reason rather than by design: one sits in a function declared with `(` and so runs in a subshell, the other in a brace function no @test calls. That boundary is stated in the header rather than papered over. The guard moves to hack/bats-no-exit-trap.bats: its subject is every unit suite under hack/, not the report collector it grew up in. Its fixture helpers assemble the trap keyword and the signal from separate arguments, because the guard scans its own source and a fixture written as a literal would be counted as a real trap in it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7e2db4da7e |
fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and the rest -- nodes, namespaces, events, HelmReleases, packages, packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads and every module gate -- read with none. The tarball is written on the collector's last line, so a single read that never returns does not truncate the report, it loses it, on the run where it is the only evidence left. Each read now goes through the shape that fits it: the existing bounded reader for a read whose output is a file, the existing selector for one that feeds a loop, and a new probe for the `if kubectl get deploy ...` gates. A gate is a read like any other, and an unbounded one is worse than the reads it guards, since the module below it never starts. Its cut-off is recorded, because skipping the module leaves the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. The reads that stay on the bare wrapper say so where they sit; what they share is that the output is not the reader's to write, going into a variable or through a pipeline that transforms it, so there is no moment at which a marker could attach. The header claimed a per-section list of what was bounded, and the tests froze that list plus a per-section count of the unbounded reads. Both are gone. The list had already gone stale, naming four sections while the file had ten, and the counts were an undercount by construction -- they matched a redirect to `$VAR/`, which no gate, selector or command substitution has. One test replaces both: no read below the library guard runs without a ceiling, checked by looking for a counterexample. A ceiling per read is not a ceiling on the report, and the header now says so where it used to imply the opposite. The pod, VM, VMI and services walks carry a count cap and a time budget; the walks over nodes, namespaces, HelmReleases, packages, PVCs and certificates carry neither, so their cost is the cluster's to set. What this removes is the read that never returns. The bounded reader now names the tool it ran and describes what that tool returned. It served kubectl alone until this change gave it talosctl and the four host commands, so a `df` that hit a stale mount left a marker reading "kubectl exited 1 ... not because the object ended" -- two false statements in one line, in an artifact whose premise is that its notes can be believed. The exec reader keeps the remote command's whole message in the listing it belongs to. `kubectl exec` appends its own `command terminated with exit code N` after whatever the command said, so quoting the last line there drops the cause and keeps the trailer, and sending the message away from the file leaves a zero-byte listing -- the "returned nothing, or never ran" question this collector exists to answer. Its cut-off note also stops asserting that the archive is beside it: a stream that got members across is kept and truncated, one killed before its first byte leaves zero bytes and the call site's emptiness check removes it, and the reader runs before that check, so it names both. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
920ad3e30d |
fix(e2e): bound the report reads that run inside a pod
The LINSTOR module runs six reads through `kubectl exec`: three listings, the error-report index, and the controller and per-satellite ErrorReport bundles. They are the only reads in the collector whose duration is set by a workload rather than by the apiserver -- a wedged controller, a satellite that stops answering, or a broken SPDY upgrade holds the stream open with no clock anywhere in the path. The tarball is written on the collector's last line, so one of those never returning does not truncate the report, it loses it. They go through a reader of their own rather than the existing one. `kubectl exec` returns the remote command's exit status, so a 124 there is either this collector's deadline expiring or a command that itself exited 124, and the status does not separate them; describing it the way every `get`, `logs` and `describe` is described would state a mechanism about the machine writing the report for something that may have happened inside a pod. The note says what is open instead of resolving it, and goes beside the output rather than into it, since two of the six stream a gzipped tarball that a line of prose turns into an archive nothing can open. kubectl's own message travels with the note for the same reason -- until now it was merged into the listings, so every run put the "Defaulted container" line at the top of a table. The guard that held the old assumption asserted only that no bounded read was an exec, which leaving every exec unbounded satisfies. It now also requires each exec to be bounded and requires the reader not to borrow the classification written for reads whose 124 is unambiguous. Matching on the verb rather than on the string `kubectl exec` is part of the fix: half of these calls spell it `kubectl -n cozy-linstor exec`, which that string never saw. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
96e33fe656 |
ci(e2e): give the report collection step a ceiling of its own
The step that runs hack/cozyreport.sh carried no timeout-minutes in any of the three workflows that run it, so it inherited only the job's 180. The collector writes its tarball on its last line and the upload is the next step, so a read that never returns spends what is left of the job budget, the job is cancelled, and the upload never runs -- the artifact is lost entirely rather than truncated, on exactly the run where it is the only evidence left. `|| true` guards against a non-zero exit, not against a read that does not return. 30 minutes is measured against the step rather than divided out of the job: it takes 3-42s across sampled runs, on failed e2e jobs as well as green ones, and 30 leaves five sixths of the job budget for the steps that upload what this one produced. A test pins the ceiling in all three workflows and requires it to stay under the job's own, since a step ceiling at or above the job's cap is the same as no ceiling at all. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
c78326281e |
fix(tests): freeze the EXIT-trap count for run-kubernetes-talos-diagnostics_test.bats
hack/run-kubernetes-talos-diagnostics_test.bats arrived carrying eight EXIT traps and was not added to the frozen set, so `found` and `frozen` disagree by exactly that one entry and the guard fails on every branch cut from main. Updating the count is what the comment above the guard prescribes for a file that did not exist when the freeze was written. The guard did its job: it noticed. The cost is wider than one red check. hack/cozytest.sh stops at the first failing test in a file, and the guard is test 83 of 166 in cozyreport.bats, so the remaining 83 do not run. The Makefile loop is `for f in $(BATS_UNIT_FILES); do hack/cozytest.sh "$f" || exit 1; done`, which aborts the whole loop, and cozyreport.bats sorts fourteenth of forty-two, so twenty-eight files do not run either. Downstream of that, finalize needs checks, and the e2e job needs finalize, so E2E Tests is skipped rather than executed. Verified by running the file to completion with bats, which walks past a failure unlike cozytest.sh: 166 of 166 pass with this entry present. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Mattia Eleuteri
|
3aef42b993 |
fix(tests): freeze the EXIT-trap count for multus-install-cni-plugins.bats
hack/multus-install-cni-plugins.bats arrived on main in
|
||
|
Aleksei Sviridkin
|
e80b8e370f |
fix(e2e): collect logs for unready pods and record truncated captures
The failure snapshot selected pods by their STATUS column, so a pod whose container runs while its readiness probe never passes -- STATUS Running, READY 0/1 -- was dropped from the report entirely. That is a common way for a suite to time out: the replica never joins, its owning HelmRelease sits in InProgress, and the artifact carries no trace of the pod the run died on. Select on the READY column instead, which also covers the partially ready multi-container case, and keep every phase that was collected before. On one degraded cluster that is four of the seven broken pods the old filter reported nothing about. The VMI and VM sections had the same blind spot and now share one selector that reads READY as the last field, because a Pending object prints fewer columns than a Running one and the two kinds differ only in how many columns precede READY. READY counts ready containers rather than the pod's Ready condition, so a pod held back by an unsatisfied readiness gate is out of scope; a test pins that rather than leaving it to be found in an empty directory. Logs are requested for every container of a selected pod rather than skipped for Pending ones, and the per-pod collection moved into a helper so it can be driven against a stub kubectl. kubectl's refusal is the finding, and an empty result is ambiguous on its own: a read against an unscheduled pod exits 0 having printed nothing, which is what a container that started and stayed quiet also leaves. Every site that classifies a failed read now orders its branches the same way, naming the timeout first where both a cutoff and an uncapturable stderr apply. The CRD gate in the summary had no such branch at all: it returned in silence, which is byte-identical to the CRD being absent, so every section behind that gate vanished from a report whose only real problem was that the machine writing it had nowhere to put a temp file. Collection used to stop silently. A walk that hit its cap said so in the job log and nowhere in the artifact. Now the pod walk leaves COLLECTION-TRUNCATED.txt when it stops at COZY_REPORT_PODS_MAX or COZY_REPORT_PODS_BUDGET, CONTAINER-LOGS-TRUNCATED.txt on a pod cut off part way through, COLLECTION-FAILED.txt when the pod list never came back rather than the empty tree a healthy cluster produces, and COLLECTION-UNBOUNDED.txt when timeout is missing locally, which is a fact about the machine writing the report and not about the cluster. Tenant namespaces are collected first, so a cap that only bites on a broadly degraded cluster is not spent on platform pods while the pod the suite died on falls off the end. Container logs are read with --tail=2000 where they were previously read whole, and every bounded file states the bound it was written under, because --tail drops the oldest lines and nothing else in the output would say so. COZY_REPORT_POD_TAIL=-1 restores the complete log. The bounded whole-object read no longer merges stderr into pod.yaml, vm.yaml and vmi.yaml, which are read by yq rather than by eye: a warning on an otherwise complete read left an unparseable file. stderr goes to a scratch file and is copied in only when kubectl returned no object, and the truncation markers those files carry are written as YAML comments, since a bare marker line after a complete mapping is a second top-level node. Six latent faults surfaced while making these reads report. The kamaji module was gated on the LINSTOR controller, so a cluster without LINSTOR got no kamaji evidence at all, which is what a tenant-kubernetes bootstrap timeout needs most. The Flux sources section nested one jsonpath filter inside another, which kubectl rejects outright, so it printed nothing on every report ever taken and read as "all sources Ready". The HelmRelease listing took a single field as STATUS where STATUS is the Ready condition's whole message, so every failed release rendered as one word. The services section filtered for a pending LoadBalancer address on the CLUSTER-IP column, which never carries that value. The cozystack module read its image from a Deployment the installer stopped shipping when it moved to the operator, putting kubectl's NotFound in the first file of the report. And the cozystack apps section gated all three of its kinds on a CRD in an aggregated API group, so it never ran once; two of those names were wrong, and both are collected again. The tests avoid EXIT-trap cleanup, and e2e-testing.md says why: a bats test that installs one and then fails prints no TAP line at all, so output grepped for `not ok` reads as green, while hack/cozytest.sh reports the same failure correctly. A guard enforces that over a named list of files rather than a glob with exceptions, because a list of exceptions is the same claim written so that forgetting to shorten it makes it false. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |