mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-08 21:14:56 +09:00
f74546db5ff024d342c63fc5d1bb07d32c28ea4c
5690
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Myasnikov Daniil
|
f74546db5f |
ci(e2e): make the upgrade-e2e label opt-in actually start a run
main gained a `labeled` pull_request trigger whose plan/resolve guards
discard every label event except `full-e2e`. This lane's documented opt-in
("any PR a maintainer opts in with the upgrade-e2e label") therefore did
nothing after the rebase: adding the label fired a `labeled` event, `plan`
skipped, `finalize` skipped with it, and upgrade-e2e never satisfied its
needs. Before the rebase there was no `labeled` trigger at all, so the label
was only ever read on push — the gap appeared with the new base, not here.
Admit `upgrade-e2e` alongside `full-e2e` in both allowlists so labelling an
open PR starts a run, and use !cancelled() rather than always() in the job
gate, matching the sibling `e2e` job: a cancelled run must not go on to
occupy a 32-vCPU runner for up to 180 minutes.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
638cc751e2 |
ci(e2e): run the shared e2e composites in the workspace, not /tmp
The composites extracted here were written against the pre-78fc3f917 e2e job, which copied the checkout to /tmp/$SANDBOX_NAME, ran every step from there and removed it on teardown. main has since dropped that workspace: e2e is a single job on an ephemeral runner, so there is no cross-job rendezvous and the checkout is never reused. Resolving the rebase in favour of the composites would have reintroduced it, including the unguarded `rm -rf /tmp/$SANDBOX_NAME` that commit deliberately deleted rather than guarded — when an upstream step fails the unconditional `Set sandbox ID` is skipped, SANDBOX_NAME stays empty, and the `if: always()` removal expands to `rm -rf /tmp/`. Drop the Prepare/Remove workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes from e2e-prepare, e2e-collect and e2e-teardown, and point the collect and chainsaw-report upload paths at _out/. Same treatment for the upgrade-e2e job's own two `cd` prefixes. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
c9d3e4a88c |
test(e2e): skip etcd-adoption backup in the upgrade lane, keep seaweedfs baseline
The etcd v1alpha2 adoption snapshot cannot succeed in an e2e sandbox: it needs a trusted-cert (ACME) external S3 endpoint, which a sandbox (example.org, no DNS/ACME, self-signed in-cluster cert) cannot provide — matching the dev10 findings. Set migrations.etcdAdoptSkipBackup=true in the platform Package so migration 50 runs the real adoption without the infeasible snapshot. Keep seaweedfs on the root tenant (the historical default install-cozystack.bats also uses) for a realistic baseline, but drop the fragile cozy-backups-creds wait — the adoption no longer depends on the seaweedfs S3/bucket/creds chain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
52e30c626d |
ci(e2e): keep sandbox-id write out of the composite (zizmor github-env)
Pre-commit's zizmor audit flags any `>> $GITHUB_ENV` write inside a composite action as a cross-boundary env write (github-env, "may allow code execution") — even the identical line passes in a workflow file. Move the SANDBOX_NAME write back into each job as a one-liner (the pre-composite, zizmor-clean form) and have the e2e-prepare composite only copy the workspace + run prepare-env. GITHUB_JOB keeps the e2e and upgrade-e2e sandboxes distinct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
8e2543758e |
test(e2e): fix upgrade canary DB access (verified on a live cluster)
Dev-cluster validation of the seed/verify canary commands surfaced two real bugs, now fixed and re-verified (postgres=3, mariadb=3 rows): - postgres: the seed writes SQL to psql over a heredoc on stdin, but `kubectl exec` without `-i` does not forward stdin, so CREATE/INSERT silently never ran. Add `-i`. - mariadb: the nano server container ships the `mariadb` client (MariaDB 11 dropped `mysql`) and does not bind 127.0.0.1 — connecting there fails and its retry OOM-kills the container. Use the `mariadb` client against the mariadb-test-primary Service (writable primary) from mariadb-test-0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
1f6e59d19a |
test(e2e): add chainsaw-native release upgrade testing lane
Reimplements the intent of #2401 (previous-minor-stable -> current upgrade test) over the post-Chainsaw-migration codebase. The old #2401 was a single 895-line BATS file, now obsolete; this builds a two-phase Chainsaw lane bracketing an external helm upgrade, reusing existing files throughout. Flow (make upgrade-cozystack, in the e2e sandbox): install-previous (bats, prev OCI chart) -> seed (chainsaw, skipDelete) -> helm upgrade to in-tree chart (bats) -> verify (chainsaw, assert-only). DRY over existing files: - 4 composite actions (.github/actions/e2e-{download-assets,prepare,collect, teardown}) shared by both the e2e and new upgrade-e2e jobs; the e2e job is refactored onto them (behavior-preserving). - hack/e2e-wait-hr-ready.sh: the all-HR-Ready gate extracted from e2e-install-cozystack.bats so install and upgrade share one legible gate. - hack/upgrade-prev-version.sh (+ unit test) resolves the baseline version. - seed suites apply the existing app manifests by relative path (no new CRs). Trigger: an upgrade-e2e job in pull-requests.yaml, parallel to e2e, gated on the release label or a maintainer upgrade-e2e label. Advisory / non-blocking. Coverage: postgres + mariadb canary rows, redis/VM volume survival, migration stamp advance, all-HRs-Ready + no-new-unbound-PV. The tenant Kubernetes survival suites (#2931 path) are env-gated OFF (UPGRADE_E2E_TENANT_K8S) as the flakiest surface, pending dev-stand validation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
4689b2b844 |
feat(kubeovn-webhook): add client-certificate verification, off by default (#2698)
## What this PR does Adds client-certificate verification to the kube-ovn webhook, and turns it on only when an operator supplies a CA bundle. `--client-ca-file` is new and empty by default. With it empty the server sets `tls.RequestClientCert`: it asks callers for a certificate, records the issuer of whatever arrives, and serves the request either way. Set the flag and the server switches to `tls.RequireAndVerifyClientCert`, so a caller without a certificate signed by that CA is refused at the TLS handshake. **This is step one of two, and on its own it does not close [GHSA-g883-q79m-8225](https://github.com/cozystack/cozystack/security/advisories/GHSA-g883-q79m-8225).** With the default empty value nothing is rejected — the webhook still answers an unauthenticated caller, and the only change is that the caller's issuer now appears in the logs. That is deliberate: switching straight to `RequireAndVerifyClientCert` would break every cluster whose API server sends no client certificate, and there is no way to know from here which those are. The logs are how we find out. Step two, in a follow-up: read a day of issuer logs on a Talos cluster and on `isp-full-generic`, confirm the API server presents a certificate we can pin, then set the CA bundle so verification becomes mandatory. The advisory should be published as fixed only after that lands — not with this PR. An earlier revision also carried a NetworkPolicy restricting ingress to `kube-system`. It was dropped, and the title has been corrected to match; only the TLS path is left here. ### Screenshots N/A — no user-facing or UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the trigger map against the diff: no row matches. The change touches one system package and the Go image it builds — no app package, no `values.schema.json`, no `ApplicationDefinition`, no platform variant, no node prerequisite. ### Release note ```release-note feat(kubeovn-webhook): the webhook now accepts --client-ca-file. Left empty (the default) it requests and logs client certificates without requiring them; set to a CA bundle it rejects callers that do not present a certificate signed by that CA. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional client-certificate verification for the pod mutation webhook through configurable CA settings. * Added separate rate limits for requests with and without client certificates. * Added remote-address logging and clearer verification status reporting. * **Security** * Requests continue to be forwarded while certificate observations are recorded when verification is disabled. * Serving-certificate updates remain supported; client CA changes require a webhook restart. * **Documentation** * Documented client CA configuration and current certificate rotation limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
35668dd27c |
feat(dashboard): rework the Marketplace entry point and the Admin portal (#3928)
## What this PR does Makes the Console the dashboard's entry point, reworks the Marketplace into an ordinary destination, and turns the Admin portal into the place where tenant-wide administration actually lives. **The Console is where the dashboard opens.** Visiting the dashboard without a path lands on the Console, which shows what the tenant already runs — what someone opening it almost always came for — and the header gives the Console the primary emphasis. The Marketplace is a place you go deliberately, when the intent is to add something new, and it is one click away from its own tab, from the Console overview and from the command palette. **Marketplace** opens the grouped application list directly. The "Choose a product to deploy" landing page and its Show-all-apps button are gone, and category links keep filtering to a single category. **Administration moves from Console to Admin.** Tenants, Modules, External IPs and the per-tenant Info page need no special permission, so the Admin tab is now always visible instead of being gated. The two cluster-wide operator areas keep their own independent gates, so Capacity and Backup Classes still appear only for users who can open them. Resource routes became portal-aware, so a detail, edit or order page opened from Admin stays under `/admin` instead of jumping back to `/console`. **Tenants** is a hierarchy tree instead of a flat table. It is built from `TenantNamespace` labels (`tenant.cozystack.io/<ancestor-ns>`), so a tenant whose parent is not accessible attaches to its nearest visible ancestor rather than disappearing, and rows are named relative to that parent — a sub-tenant reads as `crpjxhwm`, not `whmcs-crpjxhwm`. Each row carries its own actions: open that tenant's Info, or create a sub-tenant at any level of the tree. **Modules** is rebuilt around the same tree. It previously showed only the selected tenant's modules as Enabled or Disabled cards, which meant a module provided by a parent tenant looked disabled, and clicking it navigated to a Tenant CR that does not exist in the child's namespace — an endless spinner. The page now covers every tenant the user can access, with each tenant's modules as chips showing the module icon, the namespace running it and its readiness. Availability comes from the `namespace.cozystack.io/<module>` label on each `TenantNamespace`, which names the providing namespace: the tenant itself for a local module, an ancestor for an inherited one. An inherited chip is shown only when that ancestor is out of the user's reach, since otherwise its own chip already shows the module. Clicking any chip opens the module in whichever tenant runs it. Tenants whose whole subtree has no modules collapse into a "+ N tenants" branch. **External IPs** lists LoadBalancer services across every accessible tenant in one table with a Namespace column, instead of only the selected tenant. A namespace whose services the user cannot list contributes no rows rather than failing the page. Two navigation fixes come along with this. Detail pages offered a fixed tab set, so Info showed Workloads, Services and Ingresses tabs it never fills — tabs now appear only for resource groups the instance actually owns, probed through the tabs' own list refs so they share one cache entry and one watch. Back and the sidebar followed the resource plural rather than the entry point, so a module page sent the user to a one-item list and highlighted nothing; module pages now return to Modules and keep it highlighted, while Info returns to Tenants. Finally, copying a secret key no longer requires revealing it on screen first, and the button briefly turns into a check mark so it is visible that the value reached the clipboard. ### Screenshots <img width="1720" height="1374" alt="Screenshot 2026-08-20 at 21-08-30 Cozystack" src="https://github.com/user-attachments/assets/8499577c-8f9e-4df9-a6bb-df507497a4a0" /> <img width="1720" height="1374" alt="Screenshot 2026-08-20 at 21-08-40 Cozystack" src="https://github.com/user-attachments/assets/01d256e9-2585-41fd-8070-b910f03d8f3f" /> ### Downstream repositories The diff touches only `packages/system/dashboard/images/console`. The console is vendored in this repository (`cozystack-ui` is archived), and the change adds no package, no values schema, no `ApplicationDefinition` semantics and nothing under `hack/`, so no repository in the trigger map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(dashboard): the dashboard now opens on the Console rather than the Marketplace, the Marketplace opens the application list directly, and tenant administration moves into the Admin portal, which is now always available. Tenants and Modules are shown as the tenant hierarchy across every tenant the user can access: a module provided by a parent tenant is no longer reported as disabled, and opening it lands on the tenant that actually runs it instead of hanging. External IPs covers every accessible tenant in one table, detail pages only offer tabs the instance has resources for, and a secret key can be copied while it stays masked. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Administration now includes dedicated tenant, module, external IP, and information pages. - Tenants appear in a collapsible hierarchy with creation, editing, and detail navigation. - Modules and external IPs are shown across accessible tenant namespaces, including inherited modules. - Application details show tabs only for available resources. - Secret values can be copied while remaining masked. - Console is now the primary landing page, with Marketplace available separately. - **Bug Fixes** - Improved sidebar highlighting for nested pages. - Administration is accessible without operator permissions, while restricted areas remain protected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
724354ffe1 |
fix(console): address the review findings on the Admin portal rework
Four of the five findings share one root cause: a node in the tenant
tree can attach to its nearest *visible* ancestor when its real parent
is inaccessible, and per-row actions then derived their target from
that bridged ancestor.
The Edit action on a bridged tenant row pointed at a Tenant CR that
does not exist. The CR lives in the real parent's namespace, so the
target now comes from realParentNamespace — and Edit is offered only
when that namespace is readable, because otherwise there is no CR the
user could open and the button could only ever dead-end. The test that
asserted two Edit buttons had frozen the broken target as expected
behaviour; it now names the rows it expects, and the Edit button gained
a title so that assertion means something.
An inherited module chip is rendered only when the providing tenant is
NOT visible, yet it was a live button: clicking it denied the instance
GET and switched the active tenant to a namespace outside the visible
set, which the selector then bounced to an arbitrary fallback. It is
now an inert badge that states where the module comes from.
InfoRedirect's missing-definition branch hardcoded /console, throwing an
admin out of the portal its own docstring promises to preserve.
The sidebar's alsoMatch used a bare startsWith, so /admin/apps would
light up for /admin/apps-v2. It now matches whole path segments.
And SecretsTab let a rejected clipboard write escape as an unhandled
rejection, leaving the button silent — a denied permission or an
insecure context was indistinguishable from a no-op. It now reports the
failure.
Declined: CodeRabbit asked to split `import { useK8sList, type K8sResource }`
into a separate `import type`. The inline modifier is valid under
verbatimModuleSyntax and is what 26 other imports in this console use,
so the change would make the file the odd one out.
Reported-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
|
||
|
myasnikovdaniil
|
334e2d8bf9 |
test(e2e): record the runner's SVM feature flags in the identity capture (#3945)
Adds one line to the runner identity record, taken from the `lscpu` read it already does: ``` svm features: svm npt nrip_save vgif ``` `vgif` decides whether a guest that is itself running a hypervisor pays a VM exit on every `CLGI`/`STGI`. Here that is the sandbox Talos nodes hosting the tenant KubeVirt VMs, so it only bites at double nesting, and the sandbox nodes' own boot is unaffected. Across the archived reports no run missing the flag has passed a tenant-Kubernetes suite: 0 of 62, against 11 of 12 with it. Exposure varies between hosts of the same shape, so it is not a property of the lane and cannot be recovered once the runner is recycled. Recording it on every run, red and green, is what makes the correlation countable rather than reconstructed by hand. `apply` runs `hack/e2e-runner-identity.sh` before `docker run`, and every workflow that runs the suite reaches it through `prepare-env`, so this is captured once per run on both paths. `lscpu` is already answering there: the `cpu model` line in today's records is parsed out of the same table. No new bounded read, no compiler on the runner, no sysfs. Gated on `svm` so an Intel host's `vnmi` is not reported as an SVM feature. Five tests for the reader. Identity suite 62 assertions, `hack/` sweep 732. No behaviour change and no gating on the flag. |
||
|
Myasnikov Daniil
|
a94abe0971 |
test(e2e): record the runner's SVM feature flags in the identity capture
Adds one line to the runner identity record, taken from the lscpu read it
already does:
svm features: svm npt nrip_save vgif
vgif decides whether a guest that is itself running a hypervisor pays a VM exit
on every CLGI/STGI. Here that is the sandbox Talos nodes hosting the tenant
KubeVirt VMs, so it only bites at double nesting, and the sandbox nodes' own
boot is unaffected.
Across the archived reports no run missing the flag has passed a
tenant-Kubernetes suite: 0 of 62, against 11 of 12 with it. Exposure varies
between hosts of the same shape, so it is not a property of the lane and cannot
be recovered once the runner is recycled. Recording it on every run, red and
green, is what makes the correlation countable rather than reconstructed by hand.
No new bounded read, no compiler on the runner, no sysfs. Gated on svm so an
Intel host's vnmi is not reported as an SVM feature.
Five tests for the reader. Identity suite 62 assertions, hack/ sweep 732.
No behaviour change and no gating on the flag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Aleksei Sviridkin
|
3c780acd41 |
test(e2e): stop blocking merges on the tenant node-join deadline (#3932)
## What this PR does The tenant node-join wait in the two kubernetes e2e suites has stopped discriminating. Nested virtualisation on the shared runners degrades far enough that a worker which registers in minutes elsewhere can miss any deadline this test can afford, so a red on that one wait says as much about the machine the run landed on as about the product. What it produces in practice is a re-run, and a signal that teaches people to re-run is a signal that gets the meaningful reds re-run too. The suite still fails on it. That is the part worth reading twice, because it is what separates this from hiding a flake: the script exits non-zero, Chainsaw records a failure, and its whole catch runs (previous-instance container logs, the host snapshot, the data-plane capture, the event dump), every one of which is keyed on the test failing. The JUnit report says a test failed. What changes is one layer up. The wait writes a `SOFT-RED-node-join.txt` marker beside its diagnostics, and a new lane-side gate, `hack/e2e-node-join-soft-red.sh`, reads that marker to decide whether the job blocks. Softening inside the suite would have bought a green check by throwing all of that evidence away. The gate opens on one shape only, every failed test in the run carrying that marker, and blocks on everything else, including every way it can fail to answer the question: a report it cannot read, a failed test the report does not name, a failure with no marker, a run whose report records no failure at all. A marker is written only when `timeout` reports 124 on that one wait, so an assertion after the join, or the wait failing to run, stays an ordinary red. What the marker attributes is the test rather than the individual failure: Chainsaw writes one testcase per Test and combines that Test's step errors into its single failure element, so a second step error rides soft alongside the deadline, while a failed `catch`, `finally` or step-level `cleanup` rides soft for a different reason worth keeping apart, since at the pinned version those run with no step report and never reach the failure element at all. The convention doc states both rather than working around the granularity, since the suite stops at the wait and nothing it exists to prove runs after the marker. Only the lanes that block a merge ask the gate. `pull-requests` and `e2e-fork` do; `nightly` and `e2e-tag` run the same suites and stay hard. `e2e-tag` is the release candidate's own e2e check, where a soft pass would ship a release whose tenant clusters brought up no worker nodes. `nightly` is the only place an e2e failure reaches anyone who did not open a pull request, since nothing in this tree sends a notification anywhere, so the red job is the notification. A unit guard holds that split in both directions and derives the lane set from the workflows that run the suite, so a new lane has to be placed on one side or the other before it passes. One consequence of a softened red is worth naming rather than discovering later: the job finishes green, so its result stops answering whether the suite passed, and it has two kinds of consumer that each break their own way. A step gated on the job failing silently stops running. The SSH breakpoint in `pull-requests` is that case, and it is the only interactive way into a sandbox whose node-join never finished, so the E2E step records what the gate decided and the breakpoint's condition reads it back; the step is label-gated, so the session opens for a PR someone deliberately marked for debugging rather than on every softened run. A job that reports the result silently starts reporting a pass. That is the required `E2E Tests` commit status in both pull-request lanes, and on a softened run it now says the node-join deadline was missed, that the suite failed, and that the merge is not blocked on it, instead of saying the suite passed. The handover is four links long, and a guard reads each of them inside the block it has to live in rather than anywhere in the lane: the step carrying the gate's id writes the decision to the output sink, the job that runs the suite publishes it naming that step, the job that reports the result reads it off that job, and it branches on the value ahead of the branch that would otherwise call the run a pass. A link matched file-wide passes while it sits on a neighbour, and the value is then empty for good with nothing to say so. The first link is read tighter still, and in both directions: the write has to be an unconditional statement in the same branch as the exit that makes the job green, that branch has to fall through to a written non-zero exit, no other green exit may leave the step, the step may not be marked unable to fail, and the write may not appear anywhere else. A write under a condition of its own records the decision on some runs and not others; an exit that escapes the branch turns a red the gate REFUSED to soften into a green job with nothing recorded; and a write on the passing path would report a suite that passed as one that missed the deadline. What the guard cannot do is stated with it: it cannot tell one reporter from two, it reads the branch as text rather than evaluating it, and every block it reads is delimited by indentation. What is given up is real. A regression whose only symptom is that workers never reach Ready no longer blocks a merge, and these two suites are the only ones in the tree that boot tenant worker nodes. The mechanism sees a timer, not a cause, so a deterministic node-join regression rides soft as readily as a slow runner does, and the failed job stops being the thing that opens a debugging path on its own. That is a debt rather than a design, and the way out is written down beside the convention it bends: calibrate the runner canary against real lane data, then move the tolerance under the canary's alert so the lane softens a run measured as degraded and blocks one measured as healthy. Until that calibration exists the canary cannot carry the decision, which is why it does not carry it today. The deadline goes from 18m to 29m, and the reason is bounded on purpose. 29m is this test's own budget and nothing else: its clock starts where the wait starts, after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, each with a ceiling of its own, while the chart's timers have been running since the Machine was created. The autoscaler's `maxNodeProvisionTime` measures from that creation and the MachineHealthCheck's `nodeStartupTimeout` from the later of creation and `InfrastructureReady`, restarting at that transition, so the offset is not fixed and no ordering against either timer is claimed. What the chart's 20m and 30m give is a scale to pick against: a budget over the first leaves room for a remediation cycle to fit inside the window, which is a possibility rather than a guarantee, since a run can spend the whole window on one worker that never comes back. The operation ceiling both suites give the script rises with the deadline (50m to 67m), and the diagnostics phase budget rises inside it (7m to 8m), because the collectors gated ahead of the serial console cost more between them than the old budget covered. Both budget guards read their terms from the sources that set them rather than restating them. The e2e job cap rises from 180 to 215 minutes in all four lanes, and its guard is honest about what it can hold: it derives the pair's operation and catch ceilings from source and requires one figure across every lane that runs the suite, and it records that the suites' full summed ceilings (226 minutes counting the `finally` legs and the storageclass-fallback Test) have never fit any cap in that file, including at 180. The margin above the derivable bound is empirical instead: three nightlies that failed both bringup Tests ran their full-suite e2e job in 149, 158 and 160 minutes under the previous ceilings, a run of that shape gains at most 12 minutes per suite here, and 215 clears the resulting projection by about 30 minutes where 180 cleared its measurement by 20. The prose that restates the cap outside the workflows moves with it, in the Chainsaw config and in the two report collectors that derive their own headroom from it. Test coverage: a new unit suite for the gate pins the direction it fails in (an unreadable report, a failed testcase the report leaves unnamed or names with the empty string or with blanks only, a failure without a marker, and a report recording no failure all block), the lane split in both directions, the four-link handover of the softening decision, and the job cap. The fixtures put the unnameable case ahead of a named suite on purpose, because an awk record runs to the next testcase and a case placed last cannot exercise the leak the tally exists to catch. The existing node-join suite gains guards that the marker is written only under the status meaning the deadline expired, that it is written ahead of the diagnostics phase rather than behind it, and that the deadline quoted in prose across the tree is the one the wait gives, in words as well as in minutes. Every one of those guards was checked by mutating the thing it protects and confirming it, and only it, goes red. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): the tenant node-join deadline still fails its suite and still collects its diagnostics, but no longer blocks a merge in the pull-request lanes; nightly and the release-tag lane stay hard ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - E2E checks can now report a successful, non-blocking status when the only failure is the recognized tenant node-join deadline. - Failure diagnostics and suite results remain available for these tolerated failures. - **Bug Fixes** - Improved detection and attribution of node-join deadline failures, avoiding false-positive soft passes. - **Documentation** - Updated E2E timing, diagnostic budgets, and failure-handling guidance to reflect the longer test windows. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
ad62a8d62b |
test(e2e): stop blocking merges on the tenant node-join deadline
The tenant node-join wait no longer discriminates. Nested virtualisation on the shared runners degrades far enough that a guest which registers in minutes on real hardware can miss any deadline this test can afford, so a red on that one wait says as much about the machine the run landed on as about the product, and it teaches a reader to re-run the job. The suite still fails on it. That is what keeps Chainsaw's catch running: the previous-instance container logs, the host snapshot, the data-plane capture and the event dump are all keyed on the test failing, and the report stays honest. What changes is one layer up. The wait writes a marker beside its diagnostics, and a new lane-side gate reads that marker to decide whether the job blocks. It opens only when every failed test in the run carries one, and blocks on anything it cannot attribute: a report it could not read, a failed test the report does not name, a failure with no marker. Only the lanes that block a merge ask it. The two pull-request lanes do; nightly and the tag lane run the same suites and stay hard. A soft pass on the tag lane would ship a release candidate whose tenant clusters brought up no worker nodes, and nightly is the only place an e2e failure reaches anyone who did not open a pull request, since no lane in this tree notifies anywhere. A softened red finishes the job green, so the job's result stops answering whether the suite passed, and both kinds of consumer are handed the decision instead of inferring it. A step gated on the job failing would silently stop running: the SSH breakpoint is that case, and it still opens for a pull request marked for debugging. A job that reports the result would silently start reporting a pass: the required commit status now says the deadline was missed and that the merge is not blocked on it, rather than that the suite passed. The deadline goes to 29m. That is the test's own budget and nothing more: its clock starts after the machinedeployment wait, the LoadBalancer assignment and the healthz probe, while the chart's autoscaler and health-check timers have been running since the Machine was created, so no ordering against them is claimed. The chart's 20m and 30m give a scale to pick against, and a budget over the first leaves room for a remediation cycle without promising that one fits. The operation ceiling both suites give the script rises with the deadline, and the diagnostics phase budget rises inside it, because the collectors gated ahead of the serial console cost more between them than the old budget covered. Both budget guards read their terms from the sources that set them. The e2e job cap rises with them, and its guard holds what a guard can hold: the pair's operation and catch ceilings, and one figure across every lane that runs the suite. It also records that the suites' full summed ceilings have never fit any cap in that file, so the margin above the bound is taken from a measured run instead. The prose that restates the cap elsewhere moves with it, and the sweep that holds the deadline's restatements now reads the figure spelled in words as well as in minutes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Andrei Kvapil
|
bfd4ea0643 |
feat(dashboard): land on the Console instead of the Marketplace
Opening the dashboard without a path sent the user to the catalog, and the header gave Marketplace the primary emphasis. The Console is the better entry point: it shows what the tenant already runs, which is what someone opening the dashboard almost always came for. The Marketplace stays one click away for when the intent is to add something new. The landing path becomes a named constant beside the portal helper, so the redirect and the tab emphasis cannot drift apart, and the library's own default tab set is reordered to match rather than contradict its only consumer. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
cfd5444f16 |
fix(dashboard): let a masked secret be copied and confirm the copy
Copying a secret key required revealing it on screen first, and the button gave no sign that anything reached the clipboard. The value the list already carries is enough to copy, so the button works while the key stays masked, and it briefly turns into a check mark to confirm. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
e75500682c |
fix(dashboard): tie navigation to where a page was opened from
Detail pages offered a fixed tab set, so Info showed Workloads, Services and Ingresses tabs it never fills; tabs now appear only for resource groups the instance actually owns. The probes reuse the tabs' own list refs and label selector, so they share one cache entry and one watch. Back and the sidebar followed the plural instead of the entry point: a module page sent the user to a one-item list and lit nothing in the sidebar. Module pages now return to Modules and keep it highlighted, while Info returns to Tenants and highlights that, matching the trees they are reached from. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
1279652b56 |
feat(dashboard): list External IPs across every accessible tenant
The page showed LoadBalancer services of the selected tenant only, so an operator had to switch tenants one by one to find an address. It now lists them for every visible TenantNamespace in a single table with a Namespace column. A namespace whose services the user cannot list contributes no rows instead of failing the page. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
21b8a0e2f0 |
feat(dashboard): rebuild Modules as a tenant tree of module chips
The Modules page listed the selected tenant's modules as Enabled or Disabled cards, so a module a parent tenant provides looked disabled and clicking it led to a Tenant CR that does not exist in the child's namespace — an endless spinner. It now covers every tenant the user can access, as the same hierarchy tree the Tenants page draws, with each tenant's modules as chips carrying the module icon, the namespace running it and its readiness. Availability comes from the `namespace.cozystack.io/<module>` label on each TenantNamespace, which names the providing namespace: the tenant itself for a local module, an ancestor for an inherited one. An inherited chip appears only when that ancestor is out of the user's reach, since otherwise its own chip already shows the module. Clicking any chip opens the module in whichever tenant runs it. Tenants whose whole subtree has no modules collapse into a "+ N tenants" branch, and the Info module is left out entirely as every tenant carries it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
7a48d455ac |
feat(dashboard): show the tenant hierarchy on the Tenants page
The flat table becomes a tree built from TenantNamespaces, whose `tenant.cozystack.io/<ancestor-ns>` labels carry the full ancestry. A tenant whose parent is not accessible attaches to its nearest visible ancestor instead of disappearing, and rows are labelled relative to that parent, so a sub-tenant reads as `crpjxhwm` rather than `whmcs-crpjxhwm`. Subtrees collapse from the row itself. Each row also carries its own actions: Info opens that tenant's Info page, and Create Tenant orders a sub-tenant at any level of the tree. Editing a tenant switches to the namespace holding its Tenant CR — the parent's, or its own for the hierarchy root. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
ef73a0e47c |
feat(dashboard): move Administration into the Admin portal
Administration (Tenants, Modules, External IPs, per-tenant Info) leaves the Console sidebar for Admin, which is now always visible: those pages need no special permission. The two cluster-wide operator areas keep their own gates, so Capacity and Backup Classes still appear only for users who can open them. Resource routes become portal-aware through a shared base-path helper, so a detail, edit or order page opened from Admin stays under /admin instead of jumping back to /console. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
5178e169b1 |
feat(dashboard): drop the Marketplace landing page
Open the grouped application list right away instead of the "Choose a product to deploy" page and its Show-all-apps button. Category links keep filtering to a single category. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
9c7bfb830b |
ci(workflows): make the E2E breakpoint fully label opt-in (#3926)
## What this PR does The SSH debug breakpoint that opens on a failed end-to-end run used to open automatically for pull requests authored by any project maintainer, even when nobody intended to use it, in addition to opening for pull requests carrying an explicit label. Every red end-to-end run on such a pull request held an ephemeral CI runner open for up to twenty minutes for a breakpoint nobody attached to, which adds up across an ordinary day of pull requests. This change makes the breakpoint strictly opt-in: it now opens only when the pull request carries the debug label or the release label. A maintainer who wants to attach adds the debug label before or after the run fails. The step's explanatory comment is trimmed to match: the paragraph describing the removed authorship path, including a claim about runner selection that no longer applied to the current workflow, is removed rather than rewritten. ### Screenshots Not applicable: this is a CI workflow text change with no user interface. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` |
||
|
Aleksei Sviridkin
|
61f7939ab7 |
ci(workflows): make E2E breakpoint opt-in via labels only
Drop the author-association branch from the "Breakpoint on E2E failure" step condition, so a red end-to-end run only opens an SSH breakpoint when the pull request carries the debug or release label. The dropped branch held an ephemeral CI runner open for up to twenty minutes on every red maintainer-authored pull request, whether or not anyone intended to attach. Trim the step's explanatory comment to match: drop the paragraph about the removed authorship path, including its stale claim that only the debug label affects runner selection, since this workflow has no conditional runs-on to begin with. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
4ebf9c25b4 |
test(e2e): capture CertificateSigningRequests in cozyreport (#3925)
## What this PR does hack/cozyreport.sh never captured CertificateSigningRequests, so the hypothesis that a node join hangs on kubelet certificate issuance (bootstrap token, CSR, approval) had no artifact to check it against. This adds a bounded CSR list and a per object yaml capture for the host cluster, next to the existing node walk, plus the same capture for every tenant Kubernetes cluster whose kubeconfig a failed kubernetes-* Chainsaw test left on disk under tenantkubeconfig-<test>. Both captures reuse the existing bounded read helpers (cozyreport_read_object, cozyreport_select_objects), so a cut off or refused read leaves the same truncation and failure markers every other capture in the file already produces, and an empty CSR list is not treated as an error. The tenant walk globs tenantkubeconfig-* and skips silently when nothing matches, which is the ordinary case for every suite that is not kubernetes-* and for a kubernetes-* run that passed and cleaned up after itself. Neither walk carries a count cap or an object time budget, unlike the VM/VMI/service walks in the same section: this follows the file's own documented convention that an uncapped per-object walk is acceptable when the collection step's own ceiling is the backstop, and CSR count on a freshly provisioned e2e cluster is bounded by node count and kubelet retry frequency rather than by workload scale; if a real collected report ever shows the CSR walk taking a meaningful share of collection time, add the existing count-cap-plus-time-budget pattern here and update the header's BUDGETS arithmetic to match. A bats test pins the shape of both walks against the script source, and a second test proves the tenant walk never invokes kubectl when no tenant kubeconfig is present on disk. ### Screenshots Not applicable, this changes an e2e diagnostic collector script, not the UI. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` |
||
|
Aleksei Sviridkin
|
e6be333c06 |
test(e2e): capture CertificateSigningRequests in cozyreport
hack/cozyreport.sh never captured CertificateSigningRequests, leaving the hypothesis that a node join hangs on kubelet certificate issuance (bootstrap token, CSR, approval) unverifiable from the collected artifact. Add a bounded CSR list and a per object yaml capture for the host cluster, next to the existing node walk, and the same capture for every tenant Kubernetes cluster whose kubeconfig a failed kubernetes-* Chainsaw test left on disk. Both walks reuse the existing bounded read helpers, so a cut off or refused read produces the same truncation and failure markers every other capture in the file already writes, and an empty CSR list is not treated as an error. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Andrei Kvapil
|
2eb5374e90 |
fix(kubeovn-webhook): make the observation unforgeable and stop advertising an enable path that does not exist
Review found the observation could be silenced and the feature could not be turned on. Both are fixed here; enforcement itself stays out of scope for this step. Rate-limit the observation instead of remembering issuers. Keying on the issuer was the mistake: with RequestClientCert that string comes from the caller, so a bounded set of first-seen values can be filled with 32 fabricated issuers before the API server's first call, permanently suppressing the signal the feature exists to produce, while an unbounded one grows without limit. A rate limit cannot be starved -- a burst costs the current interval and the next real handshake is logged in the one after. The two outcomes are limited separately so a flood of one cannot hide the other, and both lines now carry the peer address, since 'none presented' fires for any certless caller and was not attributable. Say 'verified' rather than 'UNVERIFIED' once enforcement is on, where the handshake really has checked the chain. Stop telling operators to point --client-ca-file at an in-pod path. No such path exists: the pod mounts only its own serving secret, whose ca.crt is this webhook's serving CA, so every value either aborts startup or rejects the API server -- and with failurePolicy: Fail that stops pod creation cluster-wide. Say what is actually true and defer enabling to the step that adds the volume. Correct the claim in cert.go that this webhook configures no ClientCAs. It can now, that pool is read once at startup and never reloaded, and the comment justified skipping reload on the old premise. Record that reloading has to land together with the mount, since neither is usable alone. Cover the observation with tests: rate limiting, the burst it exists to survive, concurrency, both branches, the enforcing wording, and that admission is never blocked by logging. Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
07fd4d9a2c |
fix(kubeovn-webhook): bound the issuer observation and mark it untrusted
Both points came out of review of this branch, and both live in the observation path added for the staged rollout. The set of seen issuers was unbounded. With tls.RequestClientCert the server accepts whatever certificate a caller offers, so the issuer string is supplied by that caller: anyone able to open a connection could mint fresh issuers and grow the map and the log without limit. Cap it, and say once when the cap is reached -- a cap hit on a quiet cluster is itself the signal that the issuers are fabricated. The log also invited the operator to pin '--client-ca-file' to whichever CA it had just printed. That CA is unverified by construction, so an attacker who can reach the port can propose one. Say plainly in the log and in the comment that the value is a claim rather than an identity, and that it has to be confirmed out of band. Enforcement still depends only on --client-ca-file, which is trusted control-plane configuration. Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
fe724b2f8c |
fix(kubeovn-webhook): wire client-cert verification end-to-end, observe first
Review of the original commits found both hardening mechanisms inert, and that was correct: the NetworkPolicy could never be enforced because packages/system/kubeovn/values.yaml sets ENABLE_NP: false, so kube-ovn-controller runs with --enable-np=false and programs no ACLs; and --client-ca-file was never passed by the chart, so the RequireAndVerifyClientCert branch was unreachable. Only the TLS 1.2 floor did anything. The NetworkPolicy is dropped rather than fixed. Enforcing it would mean turning on ENABLE_NP for the whole cluster, which is a far larger change than this disclosure warrants, and shipping an object that looks like a control but enforces nothing is worse than not shipping it. Client-cert verification is now reachable: clientCAFile is a chart value and the flag is passed when set. It stays off by default, because turning it on blind is dangerous — this webhook is registered failurePolicy: Fail, so if the API server does not present a certificate (it only does when the cluster runs an AdmissionConfiguration naming a kubeConfigFile for this webhook), requiring one stops pod creation cluster-wide. So the default is tls.RequestClientCert: certificates are requested and logged, never required. Nothing is rejected and admission cannot break, while the log states per issuer whether enforcement is possible on this cluster. Setting clientCAFile then flips it to RequireAndVerifyClientCert. Addresses GHSA-g883-q79m-8225. This commit makes the fix deployable and observable; enforcement is a follow-up once the logs confirm the API server presents a certificate on each supported variant. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Claude
|
e05b24d742 |
fix(kubeovn-webhook): enforce TLS 1.2 minimum version
Prevent use of older, insecure TLS protocols. https://claude.ai/code/session_013DRdaUJMYxonsjVpWxJSq3 Signed-off-by: Claude <noreply@anthropic.com> |
||
|
Claude
|
c90a7b07db |
fix(kubeovn-webhook): add NetworkPolicy and mTLS support
Address unauthenticated access vulnerability (GHSA-g883-q79m-8225) by: - Adding NetworkPolicy to restrict ingress to kube-system namespace only - Adding optional mTLS support via --client-ca-file flag for client certificate verification https://claude.ai/code/session_013DRdaUJMYxonsjVpWxJSq3 Signed-off-by: Claude <noreply@anthropic.com> |
||
|
Aleksei Sviridkin
|
9ccb133b66 |
fix(ci): de-escalate select-e2e and name every escalation (#3817)
## Why
I replayed the 150 most recently merged pull requests through
`hack/select-e2e.sh` with every escalation branch instrumented. 118 of
them (78.7%) ran the full 21-suite Chainsaw run, and the causes, counted
once per escalating PR with the count where that cause was the only one
in brackets, were `FULL_PATTERN` 88 [68], `NO_SUITE_FOR_GROUP` 24 [24],
`CHAINSAW_SHARED` 18 [1] and `UNCLASSIFIED` 7 [4]. `YQ_BROKEN`,
`NO_GRAPH_OWNER` and `BACKSTOP_EMPTY` never fired.
Two problems come out of that. Most of the escalation is real but
unnecessary. Whole classes of path that provably cannot affect a
Chainsaw suite were escalating because no rule claimed them. And the
commonest cause of a full run was absent from the log, so "why did this
pull request run everything" could only be answered by re-deriving the
selection by hand.
## What changes
**Every escalation now names its cause on stderr.** Seven branches could
reach the full suite and four did it in silence, `full_suite_pattern`
among them, the commonest cause by a wide margin, so the usual answer
was the one the log never carried. The lines go to stderr and must stay
there: stdout is the suite list and both e2e lanes parse it, so a reason
line on stdout would be read as a suite name. A unit test asserts every
message, verified by muting each of the eleven in turn and confirming
the file goes red for all of them. A reason that regresses to silence
changes no selection and is otherwise invisible.
**`cozystack.etcd-application` gains its missing `dependsOn:
cozystack.etcd-operator`.** `packages/extra/etcd` renders `kind:
EtcdCluster` from `etcd-operator.cozystack.io/v1alpha2`, and this was
the only operator-backed application source in the tree with no edge to
its operator. It is a latent install bug as much as a selection gap:
without the edge `etcd-rd` registers the ApplicationDefinition as soon
as the engine is up, so a tenant can create an `Etcd` before the
operator exists and its HelmRelease fails on `no matches for kind
"EtcdCluster"`, and `hack/select-install.sh`'s forward closure for the
`etcd` suite omitted the operator for the same reason. On the selection
side `cozystack.etcd-operator` reached no runnable suite, so every
change to the operator ran all 21.
**`vm-disk-application` maps to the `vminstance` suite.** It resolved to
a `vm-disk` suite that does not exist, `intersect_suites()` dropped the
name, and the source reached nothing runnable. Neither did
`cozystack.kubevirt-cdi` above it, whose only other dependents are
`vm-default-images` and `vm-disk-application` itself, so every CDI
change ran all 21. The coverage was there and the table did not know it:
the `vminstance` suite creates a `VMDisk` and asserts the `DataVolume`
behind it (`hack/e2e-chainsaw/vminstance/vmdisk.yaml`,
`vmdisk-vmi.yaml`). Same defect class as the etcd edge above, on the
mapping side rather than the graph side, and with no production blast
radius of its own, since `src_to_suites` is read by the selector and by
nothing else.
**`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk.** Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs, which `kubevirt-cdi` states in its own
source ("Depend on cozystack-basics so the target namespace exists
first"), and that is install ordering rather than behaviour. The damage
runs opposite to the engine's: the engine fans one change out to every
app, basics narrows instead. It reaches no suite today so it escalates,
but it sits upstream of `kubevirt-cdi`, so the first suite to land under
CDI silently converts the platform's namespace-and-policy package from
the full run to that one suite. Not hypothetical — on #3426 enabling the
site-router suite takes `cozystack-basics` from 22 suites to
`site-router` alone, and nothing in the output says coverage was lost.
Dropping the reverse edges keeps the package reachable, stops it
propagating, and leaves a change to it running everything through the
per-path escalation.
**`hack/e2e-apps/<name>.bats` maps to the `<name>` suite**, off the
basename, exactly as the per-suite rule takes the name off a
`hack/e2e-chainsaw/<app>/` directory. Deliberately mapped rather than
marked inert: what remains there is wired to nothing after the Chainsaw
migration, and inert would bake that orphan status into the rule and go
quietly wrong the day a lane runs those files again. The basename is
membership-tested against the suite list on the spot, so an unmatched
one escalates there and then rather than having the verdict deferred to
the final intersection, where the rest of the diff would decide it.
**`packages/tests/` and three `.gitattributes` join
`inert_config_pattern`,** which is what the script's own header says to
do with a genuinely inert path instead of widening the fall-through.
`packages/tests/` is a helm-unittest fixture chart, and changing a test
*of* `cozy-lib` does not change `cozy-lib`, no PackageSource lists those
paths as a component, and nothing installs them, while a change to the
library itself still escalates through `packages/library/`. The
`.gitattributes` entries are enumerated rather than matched by filename,
because the justification is what those files contain (only
`linguist-generated` markers) and the name does not carry it:
`.gitattributes` can also set `filter`, `eol`, `working-tree-encoding`
and `export-subst`, each of which changes what lands in the working tree
and therefore what gets built.
**`full_suite_pattern` escalates only `hack/e2e-*.bats`, not every
`hack/*.bats`.** The root `Makefile` is the authority on the split and
draws it at exactly that prefix, `BATS_UNIT_FILES := $(filter-out
hack/e2e-%.bats,$(wildcard hack/*.bats))`, so the 60 files it keeps are
the unit lane and the e2e sandbox runs none of them, while the three it
filters out are what `packages/core/testing`'s recipes execute and those
still escalate. Being inert here does not leave them untested: `make
unit-tests` is gated on the `plan` job's `code` output, which
`pull-requests.yaml` computes as "any changed path outside `docs/`" and
never from `select-e2e.sh`, so a bats-only pull request still runs the
whole unit lane, plus install and the OpenAPI tests, since the `e2e` job
reads the same output rather than the selection.
## Measured effect
The before-and-after comparison below is a second measurement over a
slightly different population, stated separately because the two are not
interchangeable: it replays the last 150 first-parent merge commits on
`main` (`git diff --name-only $sha^1 $sha`) through the base selector
with base sources versus this branch with its own, where the histogram
above walks the 150 merged pull requests as GitHub lists them. The
populations overlap heavily and the direction is the same, and the
totals differ by two commits.
| | full suite | scoped | nothing |
|---|---|---|---|
| before | 120 | 20 | 10 |
| after | 106 | 25 | 19 |
Full-suite rate 80.0% → 70.7%, verdict changed on 15 commits. Fourteen
de-escalated and I read every one of their file lists: they are
unit-lane bats files, non-e2e workflows, `docs/`, `CODEOWNERS`,
helm-unittest fixtures and linguist markers. The fifteenth gets one
suite *wider*: a `cert-manager` change now also selects `etcd`, which is
correct, since `cert-manager` is a dependency of `etcd-operator` and the
etcd app now genuinely reaches it through the new edge.
That replay predates the two reachability rules, and replaying the same
150 first-parent merges across those two on their own — the selector at
`73bc4c0ae` against the selector at `adf231513`, same sources on both
sides — changes no verdict at all. Two of the 150 touch
`cozystack-basics` and both ran the full suite before and after, which
is the hub exclusion keeping something true rather than failing to do
anything, and none of the 150 touch `kubevirt-cdi` or `vm-disk`, so the
mapping shows up only in a direct selection.
A few individual selections, before → after:
| changed files | before | after |
|---|---|---|
| `hack/select-e2e_test.bats` | 21 suites | *nothing* |
| `hack/select-e2e_test.bats` +
`packages/apps/redis/templates/redis.yaml` | 21 suites | `redis` |
| `hack/e2e-install-cozystack.bats` | 21 suites | 21 suites |
| `packages/tests/cozy-lib-tests/**` | 21 suites | *nothing* |
| `packages/system/.gitattributes` | 21 suites | *nothing* |
| `hack/e2e-apps/postgres.bats` | 21 suites | `postgres` |
| `packages/system/etcd-operator/values.yaml` | 21 suites | `etcd` |
| `packages/system/kubevirt-cdi/values.yaml` | 21 suites | `vminstance`
|
| `packages/apps/vm-disk/values.yaml` | 21 suites | `vminstance` |
| `packages/system/cozystack-basics/values.yaml` | 21 suites | 21 suites
|
## Blast radius of the `dependsOn` edge
A `dependsOn` on a PackageSource is an install-ordering edge in
production, not only a test-selection hint, so this is the part worth
reviewing hardest. Both Packages are emitted by
`packages/core/platform/templates/bundles/system.yaml` under the single
`bundles.system.enabled` guard with no intervening conditional between
them, in every variant, so no supported configuration has the app
without the operator. `hack/select-install.sh --validate` reports the
graph still has no cycle and nothing dangling. The one new exposure is
an admin who lists `cozystack.etcd-operator` in
`bundles.disabledPackages` while keeping the app, which now leaves it
`DependenciesNotReady`. That is the same exposure postgres, mariadb,
kafka and redis already carry, with `Package.spec.ignoreDependencies` as
the escape hatch. On upgrade `etcd-rd` waits for the operator's two
HelmReleases, and an unhealthy `etcd-operator` already means broken etcd
apps.
## Testing
Every rule added or changed has a test in `hack/select-e2e_test.bats`,
the install-side change has one in `hack/select-install_test.bats`, and
the new graph edge has a helm-unittest guard at
`packages/core/platform/tests/sources_etcd_application_dependson_test.yaml`.
The round-trip test that walks every suite through both mapping tables
stays green, `vminstance` included now that two sources map onto it.
The two reachability rules were each checked by removing the rule and
confirming its own test reddens: without the mapping the CDI test falls
back to the full-suite escalation, and without the hub exclusion
`cozystack-basics` selects `redis vminstance` instead of everything. The
hub test has to seed its own hazard, since no suite sits downstream of
basics in this tree yet, so it gives `redis-application` an edge onto
basics in a copied sources directory and pins that a redis change still
selects redis, or the assertion would pass against a graph that resolved
nothing.
`make bats-unit-tests` aborts on the first failing file (`for f in …; do
… || exit 1; done`), and `hack/ghcr-mirror_test.bats` fails on `main`
today and sorts before `select-e2e_test.bats`, so the aggregate target
never reaches these tests. I ran the 60 unit files individually instead:
59 pass, and the one failure is that pre-existing one, byte-identical to
the base commit and referencing nothing this branch touches. Also green:
`make helm-unit-tests`, `make rd-presets-check migrations-target-check
test-check-readiness`, `hack/select-install.sh --validate`, and
`pre-commit run --all-files` leaving a clean tree.
## Two things a reviewer should know, neither addressed here
`assert_selection` in `hack/select-e2e_test.bats` aborts the whole file
with `exit 1` rather than failing a single test. That is the only thing
that works under `hack/cozytest.sh`, which provides no real `run`,
`$status` or `skip`, and it is the idiom the existing tests in that file
already use. It becomes wrong when #3497 and #3498 flatten these onto
real `bats(1)`, where a failing assert should end one test and let the
rest report. Worth converting with that work rather than ahead of it.
`cozystack.mongodb-application` carries the identical missing operator
edge: `packages/apps/mongodb` renders `psmdb.percona.com/v1` while the
source depends only on `cozystack.networking` and
`cozystack.cozystack-engine`, so `cozystack.mongodb-operator` still
reaches no runnable suite and every change to it escalates to all 21.
Same defect class and same fix shape as the etcd edge, but it is a
second production install-ordering change and belongs in its own pull
request.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- The default etcd application now includes its operator and required
platform dependencies, ensuring the EtcdCluster provider is available.
- **Bug Fixes**
- Improved end-to-end test selection for application-specific changes,
shared components, inert paths, and unresolved or unclassified files.
- Full-suite escalations now provide clear reasons through standard
error.
- **Tests**
- Expanded coverage for test selection, dependency reachability, and
etcd application dependency declarations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
1592fd9e7c |
fix(ci): resolve fork PRs by open-PR head SHA, not commit association (#3922)
Fork PR e2e lane never ran. `resolve` looks up PR for head sha with
`listPullRequestsAssociatedWithCommit` on base repo, and this endpoint
returns nothing for fork PR head, because fork commit reaches base repo
only as `refs/pull/<n>/head` and association index doesn't walk
`refs/pull/*`. So every fork PR failed closed on `expected exactly one
open PR for <sha>, found 0`, got red required `E2E Tests` that its
author cant clear by pushing, and `publish`, `e2e`, `report` skipped.
Endpoint works fine for same-repo heads, that is why nobody saw it -
same-repo PRs get `E2E Tests` from `pull-requests.yaml` and never enter
this workflow at all. Of 935 runs of `E2E (fork)` 930 skipped as
designed, and all 5 that reached `resolve` stopped right there (three on
`found 0`, one on genuinely failed fork build). One of those reds was
later overwritten by mirrored same-repo run, so symptom got hidden
second time.
Checked on live api: on base repo that endpoint returns 0 for head sha
of all 5 currently open fork PRs and 1 for same-repo PRs, on fork repo
it returns 1 for same commit. Listing open PRs of base repo and matching
head sha returns 1.
### Fix
```js
const openPrs = await github.paginate(
github.rest.pulls.list,
{ owner, repo, state: 'open', per_page: 100 },
);
const open = openPrs.filter(p => p.head && p.head.sha === HEAD_SHA);
```
Same trust properties as call it replaces, this is plain base-repo api
data that fork cannot forge, and asking `state: 'open'` at source keeps
old rule that closed or merged PR never counts. Exactly-one guard and
head sha match are not changed, so stacked lower PR whose head is only
contained in upper branch still passes, and two PRs really sitting at
same head still fail closed. Pagination becomes load bearing here, repo
has 180 open PRs and fork PR that surfaced this sits on page two.
Most of diff is the comment block, rewritten so association call doesn't
come back later as an optimization.
Simulated new logic on live api, failing head resolves to exactly one
open PR and `labels` are present in list payload, so `full-e2e` override
still reads. actionlint, zizmor, `node --check` on extracted script and
pre-commit pass.
### What is not verified
`publish`, `e2e` and `report` never executed in this workflow, so first
fork PR going through the gate will exercise them for the first time. I
would watch digest push from OCI archives and `HEAD^2 == head_sha`
assert. Also there is no test harness for inline `resolve` script and
this PR doesn't add one.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved fork pull request detection by matching the exact triggering
commit.
* Prevented stacked or unrelated pull requests containing the same
commit from being incorrectly selected.
* Ensured the workflow proceeds only when exactly one matching open pull
request is found.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
bd4939ac5e |
docs(governance): describe sub-projects and third-party dependencies (#3784)
## What this PR does Brings `GOVERNANCE.md` in line with the organisation it describes: what the project governs, and what it ships but does not. ### Sub-projects The `Code Repositories` list named seven repositories and was wrong in both directions. Five of the six sub-projects whose code ships in the distribution were missing from it — `etcd-operator`, `keycloak-kms-proxy`, `local-ccm`, `cozystack-scheduler` and `ingress-nginx-with-protobuf-exporter` — while `talos-bootstrap` and `talos-meta-tool` were listed although an adopter installs neither. Nothing stated what being on the list meant, which is why it drifted. It is replaced by a section organised around the distinction that decides scope: whether a repository's output reaches the artifact an adopter installs. Packaged sub-projects get a table naming what each one contributes, its stability, and its owners — the same handles each repository's `CODEOWNERS` already carries. Independent tooling, project infrastructure and upstream forks are described in a line each, as explicitly outside that scope. Three things an evaluator finds anyway are stated rather than left to be discovered: that one fork (`apimachinery`) is compiled into the product, why, and that the fix is proposed upstream as kubernetes/kubernetes#135537; that four repositories are private, with the reason for each, and that none of them contributes to the distribution; and that telemetry is received by a component adopters do not install, with a link to the opt-out. ### Third-party Dependencies Writes down which shipped components the project does not govern. Cozystack vendors software it does not own, which is ordinary and needs no defence — but the boundary was nowhere stated. This is a Required item on the Incubation checklist ("All project metadata and resources are vendor-neutral"), and an evaluator currently has nothing in the governance document to read it off. The section says two things: what the arrangement is, and where the components are listed. It names nothing itself. The [Licenses](https://cozystack.io/docs/v1.6/operations/configuration/licenses/) page carries the list, and by its own framing carries upstreams only — Cozystack-maintained components are Apache-2.0 and are not listed there individually. Appearing on that page is therefore already the classification, so a second list here would only drift from it. Two components that belong on that page were missing from it despite shipping since v1.3.4 and v1.4.3; cozystack/website#649 adds them. Nothing moves, nothing is relicensed, no mirroring changes. This documents the existing arrangement, which each package's `values.yaml` already describes at its own level. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. It touches `GOVERNANCE.md` only, and no trigger concerns governance documents — the website triggers are package additions, platform values, variants, components, release assets, `ApplicationDefinition` semantics, the Talos pin and developer tooling. The website PR linked above is a companion, not a consequence of this diff: it fixes a pre-existing gap on the Licenses page that stands on its own. - [x] No downstream repository is affected by this change ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added governance guidance for project sub-components, including ownership, licensing, maintenance, and repository classifications. * Documented project infrastructure, upstream forks, private repositories, and telemetry services. * Clarified how third-party software is sourced, governed, mirrored, and versioned. * Linked to a license inventory listing third-party components and their applicable licenses. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
adc9c568fd |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs (#3887)
## What this PR does Break the tenant `bootstrap-token-tenant` Job out of its `helm.sh/hook: post-install,post-upgrade` lifecycle and move it into the main install phase, so the `bootstrap-token-<id>` Secret lands in the tenant `kube-system` before `MachineDeployment` can converge. ### Why it deadlocks today - Job wired as `post-install,post-upgrade` fires only after Helm's wait phase resolves - Wait phase blocks on `MachineDeployment` Ready - MD Ready blocks on nodes registering - Node registration blocks on the kubelet CSR path, which needs the very `bootstrap-token-<id>` Secret this Job produces - Result: any fresh install, or any HR reconcile that finds MD in a non-Ready state, sits in a deadlock; `HR.status.upgradeFailures` climbs without the post-upgrade hook ever getting its turn `pre-install,pre-upgrade` is not a fix either — those fire before non-hook templates apply, so `<release>-admin-kubeconfig` (materialised by Kamaji after `KamajiControlPlane` is applied) is not yet available. ### The shape that works Same shape `talos-reconcile` already uses in this chart: main-phase resources. Helm applies the SA and Job together with `KamajiControlPlane` and `talos-secrets`. Job's Pod stays pending until Kamaji publishes the admin kubeconfig Secret (`backoffLimit` covers the retry window). Job completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state — it doesn't gate it on MD. ### Verification - Reproduced the deadlock on two independent long-running tenant clusters (v1.6.x), documented in #3875 - Reproduced the manual unblock (creating the `bootstrap-token-<id>` Secret by hand in the tenant `kube-system`) on both — nodes register within seconds, MD converges within minutes, HR clears - Extended `packages/apps/kubernetes/tests/talos_templates_test.yaml` to pin the absence of `helm.sh/hook` on both rendered documents; `helm unittest` passes 9/9 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md`. The diff touches `packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml` and its test — no app add/remove, no `values.schema.json`, no core/platform values, no `hack/`, no CRD, no `ApplicationDefinition`, no telemetry metric. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): tenant `bootstrap-token-<id>` Secret is now created in the main install phase instead of a post-install/post-upgrade hook, unblocking fresh installs and any HR reconcile that finds MachineDeployment in a non-Ready state. ``` Closes #3875 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved bootstrap-token job scheduling by running it as a standard chart resource instead of a Helm lifecycle hook. * Jobs are automatically recreated when relevant configuration changes, ensuring updates are applied reliably. * Added startup handling to wait for the admin kubeconfig when needed. * Improved retry and cleanup behavior so deployments are not unnecessarily blocked. * Ensured related resources are created consistently during chart deployment. * Improved job naming stability while respecting Kubernetes naming limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6db7260b2a |
feat(kubernetes)!: remove FluxCD addon (#3379)
## What this PR does The Kubernetes app's optional `addons.fluxcd` toggle deployed a Flux Operator + FluxInstance into tenant clusters, served by two vendored Helm charts (`packages/system/fluxcd-operator`, `packages/system/fluxcd`). Those charts are **AGPL-3.0** — the last AGPL content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not depend on them: it runs classic Flux via `internal/fluxinstall`, and migration 21 already removed the Flux Operator from it. The addon defaulted to `enabled: false`. Rather than maintain an AGPL payload for an unused feature, this PR removes it entirely as a breaking change: - Delete both system packages and their `platform` source entries. - Remove the `addons.fluxcd` HelmReleases, values, Go types and generated schema/README from the Kubernetes app. - Add **migration 54**: for any tenant that had the addon enabled, the FluxCD HelmReleases in the management cluster are **suspended, stripped of their Flux finalizer and deleted**, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running inside the tenant keeps running untouched (mirrors the orphan idiom of migration 29). It runs as a `pre-upgrade` hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. After the change the only `affero`/`AGPL` matches under `packages/` are comments in `packages/system/monitoring` explaining why Grafana is not rebuilt — no AGPL payload ships. **Verification:** the Kubernetes app helm-unittests (181) and platform tests (88) pass; `make generate` was run with the CI-pinned `cozyvalues-gen v1.6.0` (schema, README, Go types, CRD); `migrations-target-check` and the `cozystack-version-stamp` bats pass; the `api/apps/v1alpha1` module builds. ### Downstream repositories This removes a user-facing addon and drops the `addons.fluxcd` field from the Kubernetes app's `values.schema.json`. Boxes are left unticked pending follow-ups (not opened from this PR): - **terraform-provider-cozystack** — models the Kubernetes app schema; the removed `addons.fluxcd` field should be checked/dropped there. - **website** — any hand-written mention of the FluxCD addon in the tenant Kubernetes docs should be removed (the managed-app reference regenerates from the README). ### Release note ```release-note feat(kubernetes)!: remove the FluxCD addon (`addons.fluxcd`). It relied on AGPL-3.0 vendored charts and was disabled by default; tenant clusters that had enabled it keep their running Flux release via an automatic upgrade migration. To remove that orphaned Flux from such a tenant, uninstall its `flux-operator`/`flux-instance` Helm releases inside the tenant cluster by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Breaking Changes** * Removed FluxCD addon support from tenant cluster configuration, including the Kubernetes tenant CRD schema, chart values schema, defaults, and published “Cluster Addons” parameters. * FluxCD-related Helm manifests are no longer rendered as part of the Kubernetes package. * FluxCD configuration is no longer accepted for new or updated tenant clusters. * **Migration** * During upgrade, existing FluxCD Helm releases are cleaned up automatically by suspending them, clearing finalizers, and deleting the management-side releases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
851a86dcf1 |
feat(platform): orphan tenant FluxCD HelmReleases on upgrade
Add migration 54 for the removed addons.fluxcd feature. For every tenant that had it enabled, the FluxCD HelmReleases in the management cluster are suspended, stripped of their Flux finalizer and deleted, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running in the tenant keeps running untouched. Runs as a pre-upgrade hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. Mirrors the orphan idiom of migration 29. Bumps migrations.targetVersion to 55. Fail closed. Dropping the finalizer is the only thing that keeps the delete inert, and migrations never re-run, so the script runs under `set -euo pipefail` with no `|| true` on the critical path and gates the delete on the finalizer actually being gone. The verify read uses `--ignore-not-found` rather than `2>/dev/null || true`, so a failed read aborts instead of reading as an empty finalizer list that passes the gate on an object whose state was never checked. A migration that stops is recoverable; one that half-orphans a fleet is not. Cover it with a docker + fake-kubectl unit test running the real script by path inside the migrations image's own alpine base (busybox ash, jq baked on), so it exercises the shipped interpreter and load-bearing pipefail. The suite pins ordering and selection (only the addon HelmReleases are touched, each suspend -> finalizer-drop -> delete in order) and the fail-closed paths (a surviving finalizer, a failing verify read, and a failing fleet scan each abort before deleting and before stamping). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
7cca3d75e7 |
feat(kubernetes)!: remove the FluxCD tenant addon
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a Flux Operator + FluxInstance into a tenant cluster. It was served by two vendored charts, packages/system/fluxcd-operator and packages/system/ fluxcd, which were the last AGPL-3.0 content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not use these charts (it runs classic Flux via internal/fluxinstall; migration 21 already removed the operator from it), and the addon defaulted to disabled. Rather than maintain an AGPL payload for an unused feature, drop it entirely: - delete both system packages and their platform source entries - remove the addons.fluxcd HelmReleases, values and generated schema from the Kubernetes app - drop the addons.fluxcd block still submitted by the e2e harness - drop the removed fluxcd package entries from CODEOWNERS BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is removed. Tenant clusters that had it enabled keep their running Flux release; see the companion migration for the upgrade path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
4385990869 |
feat(redis-operator): migrate from archived spotahome to freshworks-oss fork (#3406)
## What this PR does `spotahome/redis-operator` was archived on 2026-06-11 (read-only; last release v1.2.4, 2022). This migrates the operator to the actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5, Apache-2.0). ### Why freshworks-oss It keeps the same `RedisFailover` CRD and API group (`databases.spotahome.com`) — so existing managed Redis is unaffected — and adds a native `spec.engine: Redis|Valkey` switch, letting managed Valkey drop its Redis-compatibility shim image and run the official `valkey/valkey` image directly. ### Changes - Re-vendor the chart from `oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which carries the `engine`-aware CRD. - Build the operator image from freshworks-oss v3.3.5 source. - Re-target the label/annotation-preservation patch onto the freshworks Go module (`github.com/freshworks/redis-operator`). ### Verification - freshworks-oss source + our `labels.diff` patch compiles natively with Go 1.26.3 (`service` and `cmd` packages). - `git apply --check` of the re-targeted patch against v3.3.5 succeeds. - The operator chart renders with the cozystack-registry image override. - The image is built (freshworks-oss v3.3.5 source + our patch, multi-arch amd64/arm64), pushed to `ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in `values.yaml`. - `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats` pass with the pinned digest. ### Note for maintainers `packages/system/redis-operator` is not part of the automated `make build`; its image is built out-of-band and the digest is pinned in `values.yaml`. That image is already built and pinned in this PR; re-run `make -C packages/system/redis-operator image` only if you bump the operator version. ### Follow-up The cozystack/redis-operator fork carries cert-manager TLS work (cozystack/cozystack#2729) that freshworks-oss does not have yet; the plan is to upstream it to freshworks-oss so it lands in the shared maintained fork rather than a cozystack-only one. ### Release note ```release-note feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for injecting additional environment variables into the Redis Operator (including a default operator group id). * Added default configurable labels for the Redis Operator. * **Bug Fixes** * Preserves and merges existing Service labels and annotations during Redis Operator updates. * **Chores** * Updated the Redis Operator and Helm chart to the Freshworks-maintained 3.3.5 release, aligned chart/image sources, and simplified Helm chart retrieval. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
7ac2ffe504 |
fix(ci): resolve fork PRs by open-PR head SHA, not commit association
The fork-PR e2e lane never ran. `resolve` looked the PR up with
`listPullRequestsAssociatedWithCommit` against the base repo, and that
endpoint returns nothing for a fork PR head: the commit reaches
cozystack/cozystack only as `refs/pull/<n>/head`, and the association
index does not walk `refs/pull/*`. So every fork PR fail-closed on
"expected exactly one open PR for <sha>, found 0", stamped a red
required "E2E Tests" its author could not clear by pushing, and skipped
publish/e2e/report.
It resolves same-repo heads correctly, which is why the hole stayed
invisible — same-repo PRs get "E2E Tests" from pull-requests.yaml and
never enter this workflow. Of 935 runs, 930 skipped as designed and all
5 that reached `resolve` stopped there; three on `found 0`, and one of
those reds was later overwritten by a mirrored same-repo run, which hid
it again.
Enumerate the base repo's open pull requests instead and match on the
head SHA. Same trust properties as the call it replaces — plain
base-repo API data the fork cannot forge — and `state: 'open'` at the
source keeps the rule that a closed or merged PR never counts. The
exactly-one guard and the head_sha match are unchanged, so a stacked
lower PR still passes and two PRs genuinely at the same head still fail
closed. Pagination stops being a corner case and becomes load-bearing:
the repo has 180 open PRs, and the fork PR that surfaced this sits on
page two.
Verified against the live API for the failing case: PR #1982 at
|
||
|
IvanHunters
|
fd8d378b73 |
fix(kubernetes): rotate bootstrap-token Job on either token half; review nits
Fold bootstrapTokenSecret into the Job content hash alongside bootstrapTokenId. talos-secrets.yaml regenerates the two token halves independently, so hashing only the id left a rotated secret no-oping a completed Job and the tenant stuck on the stale bootstrap-token. Also address review feedback: drop a review-round reference from the tracked test, correct a stale hash in a test comment, pin the alternate-image hash case exactly instead of by regex, and fix a comment indent in talos-reconcile-job.yaml. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
035bde1920 |
fix(kubernetes): correct bootstrap-token Job rationale and make it churn-free
Address review of the main-phase bootstrap-token Job. The deadlock the header claimed cannot occur on this release: the kubernetes cozyrd carries helm-install-disable-wait, which cozystack-api maps to Install.DisableWait and Upgrade.DisableWait, so there is no main-resource wait phase in front of the hooks. Rewrite the rationale around the real reason to leave the hook shape: a post-upgrade hook never runs on a release whose upgrade keeps failing, so the token Secret never lands. Fix the stale 'stays a post-install hook' note in talos-reconcile-job.yaml and the 'fails the chart upgrade loudly' comment, which is untrue once Helm no longer waits on the Job. Give the immutable Job a content-hash name suffix over its whole rendered spec, mirroring talos-reconcile: a static name would fail an upgrade that lands while the Job is live (or inside its TTL) with 'spec.template: field is immutable'. Fold the looked-up bootstrapTokenId into the hash so a rotated token also rolls the Job instead of Helm no-oping a completed one. Make the admin-kubeconfig volume optional and gate the Pod on the shared wait-for-kubeconfig init container, so it waits in init (uncounted by backoffLimit) instead of FailedMount at t0. Extend the template tests for the hashed name, its rotation on an image change, and the optional volume + init container. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
542ab899c2 |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs
The bootstrap-token-tenant Job was wired as a Helm post-install/post-upgrade hook. post-* hooks fire only after Helm's wait phase resolves, and the tenant kubernetes release wait blocks on MachineDeployment Ready, which blocks on nodes registering, which needs the kubelet CSR path — for which the very bootstrap-token this Job produces has to already exist in the tenant kube-system. Fresh installs and any HR reconcile that finds MD in a non-Ready state deadlock; HR.upgradeFailures climbs while post-upgrade hooks never get their turn to run. pre-install/pre-upgrade would not help either: they fire before non-hook templates apply, so <release>-admin-kubeconfig (materialised by Kamaji after KamajiControlPlane is applied) is not yet available. Move the ServiceAccount and Job to the main install phase, matching what talos-reconcile already does in this chart. Helm applies them together with the KamajiControlPlane and talos-secrets; the Job Pod stays pending until Kamaji publishes the admin kubeconfig Secret (backoffLimit covers the retry window) and then completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state and does not gate it on MD. Extend the existing template test to pin the absence of helm.sh/hook on both the ServiceAccount and the Job, so this cannot regress silently. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
68d83cb686 |
test(e2e): add a runner-layer fixed-work canary around the node-join wait (#3919)
## What this PR does The diagnostics around the node-join wait already carry counter pairs that report time in the CPU rows, events in the KVM exits, and instantaneous values and running maxima in the files beside those. None of them is a fixed quantity of work, so none can separate a machine that was given fewer turns from a machine that took its turns and got less done with them. This adds a collector that runs two fixed amounts of work on the runner VM and times them: an integer loop and a streaming write, each under a ceiling of its own. It takes one sample before the node-join wait and one after it, on the failing and the passing path alike. Each sample is an absolute reading rather than half a difference, so one alone is still readable against the expected ranges the capture prints, and the pair says whether a slowdown began inside the interval it brackets or was already there going into it. The collector sits outside the three counter pairs that bracket the same wait rather than among them, because it occupies a core while it runs, and a burn placed inside one of those intervals would be charged to the join window it is supposed to measure. A new bats suite of 57 tests covers the collector. The existing node-join guards were extended to price it against the phase budget and to keep the spend order documented in both kubernetes suites in step with the order the block actually runs. On the budget: the sample taken before the wait is bounded at two arms of the canary ceiling plus grace, so it can add up to fifty seconds ahead of the wait, and the guard prices it at exactly that, while the guard's total stays below an honest worst case because collectors it already priced can spend more than the figures it prices them at. This branch was reviewed extensively before it was opened, and what is known to be incomplete is written down rather than left implied: the words describing how each floor sits against its healthy band are marked in the code as written rather than derived, so they are the part that can go stale when a constant moves, and the phase budget residuals are tracked in the issue the comments beside them cite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. The diff is five files: the e2e diagnostics collector library, its two new and extended bats suites, and the two chainsaw suite definitions that document the spend order. The closest call is `cozystack/ccp`, whose skills gate on `hack/package.mk` and `hack/common-envs.mk` and drive `make generate`. Neither file is touched, nothing under `hack/` is moved or renamed, and no make target changes what it does: the new suite is picked up because `BATS_UNIT_FILES` already globs `hack/*.bats`, so the file list grows and the target's contract does not. Nothing else matches. No package is added, renamed or removed, so the website app lists are unaffected. No `values.schema.json`, version enum, default or `ApplicationDefinition` changes, so the Terraform provider is unaffected. `hack/e2e-prepare-cluster.bats` is untouched, so the node prerequisites that `ansible-cozystack` and `talm` restate are unchanged. `hack/update-crd.sh`, the `release.labels` convention and the chart source kinds are untouched, so `external-apps-example` and `cozyhr` are unaffected. No label, annotation or metric name changes, so `cozy-proxy` and `cozystack-telemetry-server` are unaffected. Minimum node and network requirements are unchanged, so `examples` is unaffected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): the node-join diagnostics now time two fixed amounts of work on the runner before and after the wait, so a run that spent its wall clock without completing work is visible in the report. No user-facing change. ``` |
||
|
Aleksei Sviridkin
|
7ec5dff93a |
test(e2e): add a runner-layer fixed-work canary around the node-join wait
The counters the node-join failure path collects are read in whatever unit their source publishes: time in the CPU rows, events in the KVM exits, instantaneous values and running maxima in the files beside those. Not one of them carries a unit of work of its own, so a machine that spent every tick and got a fraction of the work done reads healthy in all of them. Add a collector that carries its own unit of work. It runs two fixed amounts of work at the runner layer and times each: a loop over a handful of integer-valued scalars, which pressure on the shared cache or the memory controller moves little, and a store stream through blocks larger than the last-level cache one core can allocate into, which is dominated by it. Fixed work gives each reading a scale of its own, so a red run can be read with no green one beside it, and the capture states the expected range and where the estimate comes from. Sampled where the three counter pairs already are, before the wait on every run and after it on both paths, but outside their brackets: it occupies a core rather than reading a file, and a burn inside one of those intervals would land in the interval each pair divides by: runner-kernel CPU time charged to the join window, and for the two counting the guests, a core taken away from what they count. Two arms, each under a twenty-second ceiling with the five-second kill grace behind it, so fifty seconds at worst against the 420s diagnostics budget. The budget guard derives that from the ceiling the source declares, and the phase-warning guard names the canary among the collectors that keep running when timeout is missing. An exit shaped like the ceiling is attributed to it only when the whole bound was on the clock: an outside kill produces the same status, and read by status alone it would publish exactly the finding this collector exists to detect. A reading below the range the legend states, and an arm the ceiling stopped before it could produce one, reach the job log from all three reporting sites, each naming where it was taken, the way a missing reading does from the two outside the diagnostics block. On the passing path the report that holds the figure is the artifact nobody downloads. Each floor is printed in the legend it comes from and sits where a bounded arm crosses it before the ceiling stops the arm, because a floor the ceiling reaches first never fires: the arm is cut off before its rate can fall that far, and every alert then arrives as a bound rather than as the slowdown the floor was meant to name. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
899529b386 |
chore(ci): add area/cilium label and map the cilium PR scope to it (#3914)
## What this PR does Adds an `area/cilium` label and routes the `cilium` PR scope to it. Cilium has more than 20 open issues of its own right now (the pinned version and its bumps, the Gateway API implementation, network policy, IPAM next to kube-ovn, LB-IPAM), and `area/networking` no longer narrows a search. #3913 is the tracking issue that uses the label. The label already exists in the repository, created by hand so the open issues could be labelled before this lands. The sync workflow runs with `delete-other-labels: false`, so nothing is at risk in between; this PR makes `labels.yml` the owner of it and updates the scope table in `docs/agents/contributing.md`. The `cilium` scope moves from `area/networking` to `area/cilium`, following the `kamaji` and `keycloak` precedent where the dedicated area replaces the broad one rather than stacking on it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Added a dedicated `area/cilium` label for Cilium-related issues and pull requests. * Updated automated labeling and contribution guidance so Cilium changes are categorized separately from general networking work. * Refined the networking label description to exclude Cilium-related items. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
467e431765 |
chore(ci): add area/cilium label and map the cilium PR scope to it
Cilium has enough open issues of its own (version pins and bumps, the Gateway API implementation, network policy, IPAM next to kube-ovn, LB-IPAM) that area/networking no longer narrows a search. Declare area/cilium in labels.yml, route the cilium scope of PR titles to it instead of area/networking, and update the scope table in the contributing guide. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Andrei Kvapil
|
0feca20e26 |
docs(governance): describe the sub-projects and what each contributes
The Code Repositories list named seven repositories and was wrong in both directions: five of the six sub-projects whose code ships in the distribution were missing from it, while talos-bootstrap and talos-meta-tool were listed although an adopter installs neither. Nothing said what being on the list meant, which is why it drifted. It is replaced by a section organised around the distinction that actually matters — whether a repository's output reaches the artifact an adopter installs. Packaged sub-projects get a table naming what each contributes, its stability and its owners; independent tooling, project infrastructure and upstream forks are described in a line each as out of that scope. Three things an evaluator will find anyway are stated rather than left to be discovered: that one fork (apimachinery) is compiled into the product and why, that four repositories are private and none of them ships, and that telemetry is received by a component adopters do not install. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
910a917115 |
docs(governance): cut the dependency section to what it has to say
The second paragraph argued that an upstream maintained by an individual is still an upstream. Nobody had disputed it, and a governance document that defends an unraised objection invites the question instead of settling it. What remains is the arrangement itself and where the components are listed. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
95e84fc185 |
docs(governance): keep the dependency list in one place
The section named two of the upstreams it describes, which put a list here and another on the Licenses page, with nothing keeping the two in step — the drift the link was meant to avoid. States the rule and the test for applying it; the components it covers stay enumerated where they already are. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
a2f7b7956c |
docs(governance): link the licenses page from the dependency section
The section states the rule; the licenses page is where the components it covers are actually enumerated, so point at it rather than restating a list that would drift. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |