Aleksei Sviridkin 0d7beaf630 fix(e2e): bound the tenant failure diagnostics per read and as a phase
The node-join diagnostics issued about a dozen kubectl reads with
neither an outer wall-clock bound nor --request-timeout, starting with a
describe nodes through the tenant kubeconfig. That read targets the
component least likely to answer in a node-join failure, and an
unbounded read there does not lose only itself: it holds the Chainsaw op
until the op is killed, so everything after it is lost rather than
truncated, the tenant crust-gather snapshot the failing exit triggers
above all.

Each read now carries the per-call bound the newer collectors on this
path already use, sized lower because this block issues a dozen back to
back where those bound one. A read that fails or is cut off says so
instead of passing in silence, and says only what was observed: a read
that did not finish is silence about the cluster, not a finding about
it. The importer walk gains a cap for the same reason its listing is now
read on its own, since a listing that never answered must not read as a
namespace with no importer in it.

Per-read bounds alone do not end the phase, and the collectors here can
together spend more than what is left of the op by the time they run.
Much of that is not ceiling-bounded at all: on the guest-Talos path the
applies, waits and deletes carry --request-timeout or --timeout, which
bound one HTTP request rather than a client retrying against a wedged
apiserver. So the phase is bounded as a phase. Once its budget is spent
the collectors that have not started are declined out loud. That is not
a worst case and is not written as one, but it does stop the snapshot
being queued behind work begun past the deadline, which is how the
snapshot was being lost. Raising the op ceiling was the alternative and
it is not one: a larger ceiling times a sum that grows with every
collector added is the same problem later.

Because what runs last is what gets declined, the block is reordered so
the cheap reads come first. The CSR list and the signer log are the
discriminator for the failure mode a follow-up fix has to target; behind
a serial-console walk and a guest-Talos capture that can spend the whole
budget between them, they were exactly what a tight run would drop.

The comments carry invariants rather than arithmetic. A comment that
states a sum has to be kept true forever in every copy of it, and any
edit to the code or the order falsifies one of them. The numbers live in
the code as values and in the guard that pins them; the prose keeps why
the budget comes from an inequality, why admission gates the start
rather than the finish, why the cheap reads run first, and which
residuals the guard's literals do not cover.

The sibling scheduling-gate branch ends in the same exit and depends on
the same snapshot, so its two reads are bounded too. So are the reads in
the talos-image-cache diagnosis, whose gate also gains
--ignore-not-found: exit 1 from kubectl get means a refused connection
or Unauthorized as readily as NotFound, and reading it as absent
announced the mirror as never deployed on the strength of a call that
never reached the apiserver. Bounding those dumps needed a note of its
own first, since timeout prints nothing when it fires and a cut-off dump
would otherwise leave a bare section header that reads as a cache with
no Pod, no Service and no EndpointSlice.

The knobs those bounds come from are validated rather than only
documented. Each is pasted somewhere that takes digits and nothing else,
and each failed differently and quietly: a suffixed or leading-zero
budget reached the arithmetic in the block's first statement and unwound
the whole function before its own headline, so a failing run produced no
diagnostics at all; a suffixed read bound became a millisecond request
timeout and made every note blame the cluster for a value; zero disabled
the bound outright; a suffixed importer cap made the comparison exit
non-zero, so the cap stopped existing; and a non-numeric grace makes
timeout exit before running the command at all. A rejected value falls
back and says so, which is what the rest of this tree does with its own
knobs. All four are re-checked where they are used, since a value set
after sourcing never passed the assignment, and the read bound is
checked again before the per-request string is composed from it so the
two halves cannot disagree. Each default is declared once as a constant,
and the re-checks read the bare variable so an unset knob is named
rather than silently defaulted.

When timeout itself is absent the reads run unbounded rather than being
bounded with a binary that is not there: the latter turns every read
into an exit 127 and every note into the cluster refusing, and collects
nothing, where unbounded is what this path did before any of this. The
phase says so once, and says what it costs rather than reassuring.

The reachability re-probe in the image-cache helper stays unbounded and
its comment names all seven of the calls it makes, the fire-and-forget
delete after the probe Pod included: the same function decides the
mirror on the happy path, so changing when it gives up is not a
diagnostic's decision to make. Its bounds are spelled at each call site
rather than precomputed, since a prefix built at source time cannot
follow a knob a test lowers afterwards while the --request-timeout
beside it does.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-09 02:51:06 +03:00
2025-10-08 09:43:34 +05:00
2025-04-01 18:48:14 +02:00
2024-02-08 12:04:32 +01:00

Cozystack Cozystack

Open Source Apache-2.0 License Support Active GitHub Release GitHub Commit OpenSSF Best Practices OpenSSF Scorecard

Cozystack

Cozystack is a free platform and framework for building clouds.

Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.

With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.

Use Cozystack to build your own cloud or provide a cost-effective development environment.

Cozystack user interface

Use-Cases

Documentation

The documentation is located on the cozystack.io website.

Read the Getting Started section for a quick start.

If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.

Versioning

Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.

Contributions

Contributions are highly appreciated and very welcomed!

In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.

You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.

If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.

Community

You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.

License

Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.

Commercial Support

A list of companies providing commercial support for this project can be found on official site.

Languages
Go 47.2%
Shell 37.6%
TypeScript 7%
Go Template 5.4%
Makefile 1.4%
Other 1.2%