mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
Aleksei Sviridkin
02a3fda471
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Extends `hack/e2e-capture-dataplane.sh`, the diagnostic collector that runs after an e2e test failure, with the instrumentation issue #3615 asks for to close its address-attribution gap. Issue #3615 found that a NotReady pod's IP can turn out to be a local address of the node's host netns instead of reachable through the ovn0 route, which makes kubelet's probes connect to the node's own loopback and fail forever. The collector already captured `ip addr show ovn0` for the affected pod's node, but nothing that could name which interface actually holds a stolen address, since it only ever looked at ovn0. This adds `ip -o addr show` (every interface) and `ip route show table local` to the same host netns capture, so the interface actually holding a local address can be identified on the next occurrence. The collector also only ever inspected NotReady pods, so a run had no same-window baseline for what a normal route and conntrack state look like on the same node. This extracts the existing per-pod capture into a reusable function and runs it a second time against one Ready, Running pod on the same node as an in-run healthy-pod control, picked while explicitly excluding hostNetwork pods (their podIP is the node's own address, so one would show the exact same local-route fingerprint as the bug for an unrelated reason). When no eligible Ready pod exists on the node, that absence is recorded in the artifact instead of left silent. This is diagnostic tooling only: it changes what gets captured after a failure, not any test's pass/fail outcome. ### Known limitations The baseline runs at a reduced scope: the host route and the kernel conntrack for the healthy pod's address, and nothing else. What it drops -- cilium CT, the OVN port binding, the OVS `ovn-installed` flag, and the pod's own Ready conditions and events -- describes how a pod was programmed, and a pod picked for being Ready and Running is programmed correctly by definition. That scope is what holds the pod path's worst case at 579s against the 600s the `cozytest.sh` trap allows; the remaining 21 seconds are what this path leaves for the LoadBalancer section that runs after it, not headroom the run has. The other caller's envelope is 300s, and the worst case does not fit it -- nor did it at the merge base, where the same walk bounds at 519s. That overrun is tracked in #3671, and the term-by-term arithmetic is in the `MAX_PODS` note in the script. The 579s above assumes the lookup memo exists. When `mktemp` fails the memo is empty, every per-node pod lookup runs live again, and the same single-pod case bounds at 667s -- past the 600s envelope as well as the 300s one. At the merge base that memo-less case bounds at 585s, inside 600s, so this change is what moves it out. The degradation is visible rather than silent: the caller's backstop truncates the capture and the run reports `data-plane capture INCOMPLETE (exit 124)`. A baseline can be present and still carry nothing. Every capture at that scope goes through the node's `kube-ovn-cni` pod, so when that lookup does not resolve, the reference section is its banner and the scope note with nothing beneath them -- and, unlike `node-<node>.txt` next to it, no line saying why. Two different standards of honesty inside one artifact. The affected-pod path has the same shape today, so this extends an existing gap rather than opening one, and it is left alone here. The per-node baseline is cached under `TMPDIR` for the run. A SIGKILL -- which is what a caller's wall-clock backstop delivers -- skips the cleanup at the end of the script and leaves that directory behind. The lookup memo this script already used has the same exposure. Nodes are visited in the order their affected pods appear in the cluster-wide list, which the API chooses. Once `MAX_PODS` cuts the walk, which pods and therefore which nodes got captured is a function of that ordering and not of anything a reader would pick. ### Screenshots Not applicable, no UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note test(e2e): capture host netns address attribution and an in-run healthy-pod reference for the dataplane failure collector ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added diagnostics for node addresses and local routing information. * Added healthy, same-node reference captures for dataplane troubleshooting, reused across affected pods. * Expanded pod diagnostics to include status, events, connection tracking, routes, network bindings, and interface state. * **Tests** * Added coverage for pod selection, retries, caching, exclusions, incomplete records, unavailable responses, and missing baselines. * Recorded diagnostic failure reasons in artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.