Aleksei Sviridkin
6ee56e3a60
## What this PR does
`hack/select-e2e.sh` escalates to the full Chainsaw suite when a changed
package reaches no suite through the dependency graph, because otherwise
that change gets no E2E coverage at all. The escalation was read off the
merged selection, so it depended on the whole diff rather than on the
path that needed it. Any other changed path contributing a suite name
made the selection non-empty and the escalation vanished with nothing in
the output to say it had. Editing a platform component together with one
unrelated suite's Chainsaw test ran that single suite instead of
everything, which is an ordinary PR shape: change a component, adjust
the e2e near it.
Coverage is now decided per changed path. Each path resolves through its
owning sources to the suites that exist, and a path reaching none of
them forces the full run whatever the rest of the diff selected. The
unit is the path rather than the source, because
`system/postgres-operator` belongs to two PackageSources where one
reaches no suite, and deciding per source would run everything on every
change to it.
The other half is the mapping that left packages unreachable from their
own suites. `src_to_suites()` covered `*-application` names plus
`external-dns`, and the final filter dropped the rest, so
`cozystack.kuberture` and `cozystack.securitygroup-controller` were
walked to and then discarded. A CNI change ran neither of the two
controller suites most likely to regress from it, and a change to either
package escalated to the full run rather than selecting the suite that
covers it. Every reached source now maps through the table, which is the
inverse of `select-install.sh`'s `suite_to_source()`; a test walks every
suite through both, so the next source name fitting neither convention
fails there instead of quietly escalating forever.
## What was measured
Behaviour was diffed against the current selector over one `values.yaml`
per package under `packages/{apps,system,extra}`: 159 packages, 8
selections change, all of them the mapping repair. Five networking
packages and `prometheus-operator-crds` widen — 19 to 21 suites and 10
to 11, both gaining `kuberture` and `securitygroup`. `kuberture` and
`securitygroup-controller` narrow from the full suite to their own
single suite. Nothing narrows that should not.
That sweep is one path per diff, so it says nothing about the per-path
escalation, which only shows on a diff with more than one path. Pairing
each package with one unrelated Chainsaw edit gives that number. 82 of
the 159 packages escalate on their own and stop escalating once the pair
is added, which is the size of the hole: for those, adding an unrelated
suite edit to the same pull request replaced the full run with one
suite. 88 packages select something different after this change, the
extra six being the mapping repair widening a selection rather than
restoring an escalation.
Worth stating alongside it, because it decides whether the branch is a
corner or the common case: 83 of the 159 escalate on a single-path diff
today, and 81 do after this change — the two that leave are `kuberture`
and `securitygroup-controller`, which now select the suite that covers
them. So the escalation is the branch most packages take, and the
swallowed one was the branch most packages took. Counting that needs an
instrument that separates a graph walk which reached everything from an
escalation which printed everything, since the two agree once the
mapping is fixed: the count above comes from a scratch suite tree
carrying one extra suite that no source names, which only an escalation
can emit.
The suite selection is what decides which tests run, so a rule that
stops firing does not go red, it disappears. The tests pin the property
rather than a worked example: that a path reaching no suite escalates
whatever else the diff holds, that a covered path never drags the full
suite in with it, and that every suite round-trips between the two
mapping tables. Each was checked by mutation — removing the rule it
guards turns it red.
## Left for a follow-up
`resolve_suites` writes into the caller's scope because POSIX sh has no
`local`, and what keeps that safe is the single call site sitting inside
`$( )`, pinned by a test that counts occurrences of the name. Declaring
the function with `( … )` instead of `{ … }` would contain those
variables by construction and let both the test and the two paragraphs
explaining the constraint go away. A count-the-name grep is the weakest
kind of pin and it exists only because the function leaks.
The output contract at the top of the script lists the causes of a
full-suite selection but not the backstop, which after this change is a
mechanism of its own: an edit under `hack/e2e-chainsaw/<dir>/` where
that directory holds no `chainsaw-test.yaml`. The behaviour is pinned by
a test and explained where it lives; only the header block is short. In
the same area, the escalation line names the owning sources rather than
the changed path, and reports the first uncovered group rather than all
of them.
## Overlap
#3434 fixes the same escalation from a different angle: it decides the
escalation from the whole source-derived selection rather than per path.
That closes the "package with no coverage plus an unrelated suite edit"
shape, and leaves "package with no coverage plus another package that
does select suites", where the source-derived selection is non-empty and
the escalation is swallowed again. There is a test for that second shape
here.
### Screenshots
Not applicable, no UI changes.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The changed files are `hack/select-e2e.sh`,
`hack/select-e2e_test.bats` and `docs/agents/e2e-testing.md`. The only
`hack/`-related trigger is cozystack/ccp, which gates on
`hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh` and on
what a make target does; nothing here is moved, renamed, or changed in
make-target behaviour. The ansible-cozystack and talm triggers are about
`hack/e2e-prepare-cluster.bats`, which is untouched.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(ci): a changed package that no E2E suite covers now runs the full suite even when the same pull request also edits an unrelated suite, and changes to the kuberture and securitygroup packages select their own suite instead of escalating
```
Closes #3330
Closes #3665
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.