mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 10:37:21 +09:00
docs/roadmap-dco
1440
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
6144805bea |
fix(mariadb): mount the chart's my.cnf under a .cnf name (#4843)
## What this PR does The MariaDB chart's my.cnf has not been read by the server since the mariadb-operator bump to v0.28.1. Up to v0.0.27 the operator mounted the referenced ConfigMap key as `/etc/mysql/conf.d/my.cnf` whatever the key was called. Since v0.0.28 the file takes the key's own name, so the chart's key `config` became `/etc/mysql/conf.d/config`. The server's `!includedir` reads only `*.cnf` files, so it skipped it, and every instance has been running on stock MariaDB defaults. On the default `t1.nano` preset that means a 128M buffer pool inside a 128Mi memory limit. This renames the key to `my.cnf` in the ConfigMap and in the MariaDB CR. The file content is unchanged. I started all four shipped versions (10.6.24, 10.11.15, 11.4.9, 11.8.5) with the rendered file under a 128m memory limit, and each one came up with the chart's values. The MySQL-only options in the file log a warning and are otherwise ignored. They and the deprecated `innodb_flush_method` are tracked in #4842. Tests: a helm-unittest suite pins the key on both sides, and the mariadb chainsaw suite gets a step that compares `@@innodb_buffer_pool_size` on the running server with the value in the rendered my.cnf. This changes behaviour on every existing instance, see the release note. Each instance rolls its pods once because the projected volume in the pod template changes: replicas first, then the primary with a switchover. After that the server runs with every setting in the chart's file. Instances created before the operator bump (May 2024) ran with these settings before. Newer ones never did. Closes #4836 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(mariadb): the chart's my.cnf is applied again. Since mariadb-operator v0.28.1 it was mounted under a name the server ignores, so MariaDB instances ran on stock defaults. Every existing instance rolls its pods once on upgrade (replicas first, then the primary with a switchover) and then runs with the chart's settings. That changes behaviour: buffer pool, redo log, connection limits and caches are sized from the instance's resources; sql_mode becomes NO_ENGINE_SUBSTITUTION, so strict mode is off and invalid values are truncated with a warning instead of rejected; innodb_flush_log_at_trx_commit=2 can lose up to about a second of committed transactions if the node crashes; innodb_fast_shutdown=0 makes every pod shutdown slower. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * MariaDB configuration is now stored and read using the `my.cnf` key. A verification check confirms the running server applies the configured buffer-pool size. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
8ab6d4948f |
test(mariadb): assert the server runs with the chart's my.cnf
The chart's unit tests can pin the ConfigMap key and the CR reference, but not what mariadb-operator does with them or which files the server then reads from /etc/mysql/conf.d. Compare innodb_buffer_pool_size as the live server reports it against the value in the rendered my.cnf, which differs from the server default on this fixture's preset. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
da28ea7ee8 |
ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU guests, the substrate the merge-gating lanes left behind. On it the tenant Kubernetes suites have missed their node-join deadline on most nights since 2026-10-02, each failure spends the whole deadline, and the job reaches its timeout before the alphabetically later suites run. Over the same days those suites passed on every PR run that selected them, where the nodes are Talos containers and a tenant worker sits one virtualization level shallower. Both lanes now use the PR lanes' container substrate. Nightly's e2e no longer waits for the nocloud disk, which it does not boot; the disk is still built and published. A manual e2e-tag dispatch against a tag that predates the container lane fails early and names the ref to dispatch from instead. DRBD, the replicated StorageClass and the Cozystack Talos node image lose their last e2e coverage with this. The docs and comments that pointed at nightly and the rc gate as that coverage now say so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
87a64071a7 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does
fixes #3740.
on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.
**what changes**
- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.
**existing clusters**
a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.
1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.
running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.
### Validation
- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at
|
||
|
Aleksei Sviridkin
|
603a4fc401 |
test(monitoring): stop the alert severity guard from missing rule sources (#4812)
## What this PR does follow-up to #3799, which added `hack/alert-severity-contract.bats`. the guard had two blind spots. it found rule files by an exact `kind: PrometheusRule` or `kind: VMRule` line, so a quoted kind or one followed by a comment took the whole file out of the check. and a rule group whose rules all come from template actions, for example an `include` of a helper, is empty once those actions are stripped, so it passed with nothing checked. discovery now also matches a quoted kind and a kind followed by a comment. a rule document without groups, or a group with no alert or recording rule left after stripping, now fails the run and names the file and the group, the same way a file that cannot be parsed already does. a group of recording rules alone still passes. on the current tree nothing changes. the wider discovery finds the same 43 files, no group comes out empty, and the guard stays green. three new cases cover a quoted kind and a commented kind with a bad severity, a group filled only by an `include`, and a group of recording rules. against the old guard the first two fail and the third passes. dropping the recording clause from the new check turns the recording case red, and the tree check with it, since the tree has recording-only groups. the suite passes under bats 1.14 with `set -u`, and the strict setup audit stays clean. ### Screenshots not a ui change. ### Downstream repositories walked the trigger map against the diff. it touches only `hack/alert-severity-contract.bats`, a unit test that no other repository restates. no entry matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(monitoring): the alert severity check now also catches rule files with a quoted or commented kind, and fails on a rule group it cannot see into instead of passing it unchecked ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Alert-rule source discovery now recognizes quoted rule kinds and kinds followed by YAML comments. * The alert severity contract now reports and rejects documents with no groups and groups with no alert or recording rules after template actions are removed. * Recording-rule-only groups continue to pass the contract. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yan Bondarenko
|
dc814f3ae5 |
test(monitoring): stop the alert severity guard from missing rule sources
The guard found rule files by an exact `kind: PrometheusRule` or `kind: VMRule` line, so a quoted kind or one followed by a comment took the file out of the check. A rule group whose rules all come from template actions is empty once those are stripped, so it passed with nothing checked. Discovery now accepts those spellings, and a rule document without groups, or a group without a visible alert or recording rule, fails the run the way a file that cannot be parsed does. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Yan Bondarenko
|
f944eeff6e |
test(hack): load the strict-mode helper in cilium-ipam-migration.bats
bats-strict-setup.bats requires every unit file to load test_helper so its tests run under set -u, and fails the whole bats lane on main because this file does not. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
84daaf0c70 |
fix(hack): keep root monitoring redirects off in the E2E install
The E2E install sets Tenant/root spec.monitoring to false, but left the platform monitoring.rootEnabled at its default of true, so cozystack-basics rendered its vlinsert/vminsert ExternalName Services into cozy-monitoring regardless. That namespace is created only by metrics-server, monitoring-agents and monitoring. A narrow suite selection disables all three, so basics fails with `namespaces "cozy-monitoring" not found` and every package waiting on it never installs. Wide selections passed only because they keep at least one of those packages. Setting rootEnabled to match the root tenant removes the dependency and is what the value's contract asks for anyway. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
0ea11b4ef0 |
test(hack): load the strict-mode helper in fork-new-images.bats
bats-strict-setup.bats requires every unit file to load test_helper so its tests run under set -u, and fails the whole bats lane on main because this file does not. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
387760b94e |
test(tests): run unit suites with Bats and retain POSIX coverage (#3849)
## What this PR does This runs the unit suite with Bats and keeps a separate `dash` pass for tests that source production shell helpers. Unit files are discovered from the filesystem, and each loads the audited helper that restores `set -u` in test bodies without changing subprocess shell options. CI pins Bats 1.14.0, keeps JUnit reports, and runs Bats for docs-only changes too. Code changes also run Helm, Go, controller and POSIX compatibility tests. Per-file Make targets and the `--keep-going` sub-make are removed because one `bats -j` run goes through all files anyway, so a red file doesn't stop the others, and `hack/bats-unit-keep-going.bats` checks this with real bats. `-k` stays in the workflow because without it a red Bats file keeps `test-controllers` from starting. Local tests require bats-core 1.5 or newer. GNU parallel is optional, and the complete local hook runs with `pre-commit run bats-unit-tests --hook-stage manual --all-files`. Live-cluster suites keep their existing runner. ### Validation `make unit-tests test-controllers -j4 -k --output-sync=target BATS_JOBS=4 BATS_POSIX_SHELL=dash` passed, including every discovered Bats file and the POSIX pass. Mutation checks reject omitted files, disabled or exported nounset, missing CI steps and missing or excessive POSIX coverage. ### Downstream repositories The developer guide update is tracked separately in [cozystack/website](https://github.com/cozystack/website/issues/717). CCP has no caller of the changed targets. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - documentation follow-up: [issue](https://github.com/cozystack/website/issues/717) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Testing & Reliability** * Bats unit tests run as a dedicated suite, with optional parallel execution and JUnit reports. * Pull request checks run Bats tests for documentation-only changes, and unit and controller tests for code changes. * POSIX-shell compatibility tests are included in the standard unit-test run and continue after individual failures. * Test reports are uploaded even when checks fail. * Added a manual local test hook for running Bats and POSIX compatibility tests. * Shared strict setup is applied across Bats unit tests. * **Documentation** * Updated testing guidance with current commands, runner compatibility, and CI coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
3e078b79a9 |
ci(e2e): refuse a new fork image before the build starts (#4767)
## What this PR does A fork PR that adds a first-party image cannot pass the fork e2e lane, and today it finds that out only after the whole build. `e2e-fork.yaml` pushes the fork's image archives only under names the base tree already exports, so a new image is refused at the push with `image '<name>' is not in the base-tree allowlist`. Nothing the contributor does on the PR changes that. This PR keeps the push allowlist as strict as it is and moves the failure to the start of the run. On fork PRs, `plan` in `pull-requests.yaml` now runs `hack/fork-new-images.sh`. The script derives the image names of the PR tree and of its base parent with the same `make -n` derivation the push step uses. If the PR tree exports a name the base does not, it fails before any build, names the images and says what has to happen: a maintainer lands the image on the base branch first, or the change runs from a branch in this repository. The check runs from the fork's own tree, so a fork can remove it. That is fine, because it only saves an honest contributor a wasted build. The push allowlist in `e2e-fork.yaml` is still the boundary. That workflow cannot call the script, since it runs from the default branch over an untrusted checkout. So `hack/fork-new-images.bats` pins the two derivations to each other, and also covers the script's behaviour and the fork-only gating of the step. I replayed the script against the merge commit of #4291 and its base parent. It fails with `cluster-api-provider-kubevirt`, the same name the push step refused there. The workflow_run lane itself cannot be exercised locally. This does not lift the limitation itself: a fork still cannot get a green run for a new image. Part of #4329. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): a fork PR that adds a new first-party image now fails in the plan job, naming the image, instead of after the whole build at the image push. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Fork pull requests that add image names not present in the base branch are now rejected during planning, with the new names reported before builds begin. * **Documentation** * Updated guidance describes the early check and clarifies how it relates to the image allowlist used during publishing. * **Tests** * Added coverage for new, unchanged, reduced, and empty image sets, along with checks that the pull request workflow applies the check only to fork submissions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
cc942fed68 |
test(tests): run unit suites with Bats and retain POSIX coverage
The shell translator rewrites test bodies and loses Bats reporting. The unit lane needs complete TAP results and JUnit reports without changing how live-cluster suites run. Sourced production helpers still need a dash pass, and unit test bodies need nounset without exporting shell options into subprocesses. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
b1c84e9c06 |
feat(platform, tests): select E2E packages and declare controller dependencies (#4022)
## What this PR does E2E installs the whole platform on a fresh cluster for every PR, whatever the PR touches, so install time and flake surface go to packages the selected suites never use. This PR makes the E2E install take only what the selected suites need. `hack/select-install.sh --disabled <suites>` generates `bundles.disabledPackages` for fresh E2E installs using an ordinary platform variant. It keeps suite owners, runtime requirements and the install baseline, then follows their forward `PackageSource` dependencies. The baseline follows the unconditional waits in `hack/e2e-install-cozystack.bats` and stays seeded directly even where the graph reaches the same packages. It also keeps platform services that no suite owns but the rest of the platform uses: `multus`, `cozystack-scheduler`, `linstor-scheduler`, `flux-plunger`, `cozy-proxy` and `ouroboros` (the Kubernetes suites assert its hairpin NAT). Backup suites also keep `backupstrategy-controller` and its dependencies, and the gateway suite keeps `kubeovn-webhook`. The E2E job now selects the suites before it installs, and passes the complement to the platform Package as `disabledPackages`. The `full-e2e` label and an empty selection keep the full install, and the nightly and tag lanes pass no list. The platform sources also change: `mongodb-application` now depends on `mongodb-operator`, and `seaweedfs-application` on `objectstorage-controller`. These dependencies apply to regular platform installs and upgrades too. The new `mongodb-operator` edge also reaches reverse test selection: a change under `packages/system/mongodb-operator` now selects the `mongodb` suite instead of falling back to the full suite. Backup APIs and the gateway webhook remain suite-specific install requirements, so they add no edges. The disable list governs fresh installs because existing Packages carry `helm.sh/resource-policy: keep`. Optional packages still need `bundles.enabledPackages`. Packages that no suite owns and the baseline does not keep stay uninstalled on purpose, also when test selection answers with every suite (monitoring exporters like `goldpinger` and `kubeovn-plunger`, apps without a suite, provider-specific operators). Nothing in the run looks at them, so installing them only adds a readiness wait, and a change to such a package gets no install check from this gate. ### Measured One run per variant on a 32 vCPU / 128 GB VM, container lane, the CI artifacts of this PR, `postgres` suite selected, 55 packages disabled: - HelmReleases 111 to 77, pods 219 to 197 - pod requests 14.8 CPU / 37.7 GiB to 13.5 CPU / 35.7 GiB - host memory peak during install 33.4 GB to 30.2 GB - `install-cozystack` 26m33s in both runs - `postgres` suite 25m50s to 25m17s, passes on the reduced install Install time did not change. The removed packages are mostly operators and apps without instances, and what the install waits on (SeaweedFS, LINSTOR, CDI, the Cluster API providers, Keycloak) stays in the baseline. With one run each, differences of a minute or two are noise. The selection, the install closure and the workflow wiring are covered by bats tests (`hack/select-install_test.bats`, `hack/select-e2e_test.bats`, `hack/platform-packages_test.bats`) and by a helm-unittest suite in the platform chart. ### Screenshots Not a UI change. ### Downstream repositories The new dependencies ship in the platform chart. No coordinated changes are needed in downstream repositories. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(platform, tests): E2E installs only the packages the selected suites need. MongoDB now depends on its operator, and SeaweedFS on the object storage controller. ``` |
||
|
Aleksei Sviridkin
|
798094e725 |
feat(release): add backport-audit, a pre-release backport triage tool (#3527)
## What this PR does
This adds `cmd/backport-audit`, a maintainer tool that checks, before a
release or an rc is cut, whether everything labelled for backport
actually landed on the release branch. It prints the URLs of the PRs
that did not and exits non-zero, so the cut can gate on it.
`docs/release.md` and `docs/agents/releasing.md` now send the patch-cut
check through it.
The docs it replaces inventoried backports with `gh pr list --search
"is:merged label:kind/backport"`. That lists every PR ever labelled, not
what reached the branch. Answering the real question by hand means
reading the branch history per PR, and that is how backports go missing
quietly.
```console
$ go build ./cmd/backport-audit
$ ./backport-audit release-1.4
=== release-1.4 === 30 candidate PRs: MISSING=5 dropped=4 backported=21
MISSING -- labelled for this line, no trace of it here (5):
https://github.com/cozystack/cozystack/pull/2704
#2704 fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig
label=kind/backport author=myasnikovdaniil merged=2026-05-26 -- no backport PR, nothing on branch
```
A `kind/backport` label does not name a branch. `backport.yaml` resolves
the target when the PR merges, so the same label means `release-1.5` on
a PR merged in June and `release-1.6` on one merged in August. The audit
reproduces that rule instead of trusting the label text. The rule
changed once, on 2026-08-03, when
|
||
|
Aleksei Sviridkin
|
5486d4ffcf |
fix(networking): harden webhook policy assumptions (#4058)
## What this PR does This is follow-up for [#3997](https://github.com/cozystack/cozystack/pull/3997). It reads ingress-nginx admission port from effective subchart values and refuses empty value, and fixes `kubeProxyReplacement=false` status because cozystack has no supported Service translator in that state. Gateway test now requires one real ClusterIP and keeps same ready Pod UID, IP and port for whole probe, so stale EndpointSlice, replaced Pod or wrong Service can't report deny. Policy selectors and two-ended deny from [#3997](https://github.com/cozystack/cozystack/pull/3997) stay unchanged, and this does not claim root cause for [#3983](https://github.com/cozystack/cozystack/issues/3983). ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Admission webhook connectivity checks now validate the configured Service and its exact backend endpoint throughout probing, improving failure detection. - Admission webhook policies now fail closed when the webhook port is not explicitly configured. - **Documentation** - Clarified that supported service routing requires `kubeProxyReplacement: true`. - Updated guidance to explain that disabling this setting or webhook policies does not provide a supported fallback. - **Tests** - Added coverage for missing webhook port configuration and strengthened networking stability checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
bcfa50d7e9 |
test(e2e): install only the packages the selected suites need
The E2E job installs the whole platform on a fresh cluster before it knows which suites will run, so every PR pays install time and flake surface for packages its suites never use. Select the suites first and pass the complement computed by select-install.sh to the platform Package as disabledPackages. The full-e2e label and an empty selection keep the full install, and the nightly and tag lanes pass no list, so they are unchanged. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f46a9797b8 |
fix(hack): keep platform services in selective installs
Dropping every package that no suite owns also drops services the rest of the platform relies on: multus for secondary networks, the cozystack and LINSTOR schedulers, flux-plunger, cozy-proxy and ouroboros, whose hairpin NAT the Kubernetes suites assert. Nothing waits for them, so no PackageSource edge reaches them, and adding one would also change reverse test selection. The remaining packages stay uninstalled on purpose, also when test selection answers with every suite: nothing in the run looks at them, so installing them only adds a readiness wait. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f863967ca5 |
fix(hack): retain the webhook used by gateway tests
The gateway namespace-churn regression requires a registered and ready pod admission webhook. Its availability cannot be tested when selective installation removes the package that registers it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
59555c895e |
fix(hack): retain controllers and APIs for selective installs
Tenant HelmReleases need a shard reconciler, and the root SeaweedFS release creates COSI resources even when no backup suite is selected. Without those controllers, a fresh reduced install cannot become ready. The Kubernetes suites create worker pools through KubernetesNodes, and MongoDB renders an operator-owned CR. Selecting only the cluster API or application chart drops resources those suites require. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
064231df47 |
fix(hack): support BWK awk in install graph validation
BWK awk rejects literal newlines in command-line assignments. Every non-empty selection and graph validation therefore aborts before it can check dependencies when this awk implementation is selected. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
361893e6f4 |
fix(hack): retain backup APIs in selective installs
Backup roundtrip suites create Bucket and backup resources that the application graph does not require. A graph-only closure therefore omits the APIs those suites need on a fresh install. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7fda27385e |
fix(hack): validate selective install graphs before output
Dangling references and cycles make the inventory unsuitable for a fresh install even when the selected suite does not reach them. Accepting that inventory would hide a broken platform graph. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
3766579e89 |
test(hack): tighten selective install assertions
PackageSource names contain dots, so regular-expression matching can accept a different source. Selecting a suite also does not prove its install closure retains every API that suite uses. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
610578be16 |
test(hack): verify selective install contract
A missing suite owner or a discarded discovery failure can leave a reduced install without the APIs its tests exercise. Graph-only checks cannot establish that the chart preserves the requested package set. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
4b13893911 |
feat(hack): generate installable selective package sets
A reduced E2E install needs the install harness prerequisites and every owner of a selected suite, including requirements absent from the application dependency graph. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
f0bd36a709 |
test(release): keep audit and workflow backport labels aligned
A label spelling accepted by only one side can make a requested backport invisible to the release check. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
Myasnikov Daniil
|
fd0a0fa29f |
test(networking): bind deny probe to live pod
Require a real admission ClusterIP and carry the EndpointSlice target UID into the direct-backend coordinate. Tie that coordinate to a non-empty current Pod snapshot so stale endpoints or masked list failures cannot produce a policy verdict. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
79e236aa7a |
test(networking): harden webhook regression proof
Probe one exact ready admission EndpointSlice backend throughout the measurement window and reject missing or ambiguous Service ports. Ensure the kubeovn-webhook Helm suites run through package discovery. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
520af57905 |
chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out internal mirror block, an internal product name, and a reviewer's name in a comment do not belong in a public repository; drop them and replace what needs a stand-in with neutral placeholders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
d2e3eb4f82 |
fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does Makes the backupcontroller test package compile again on main. #4587 added a flavor argument to `createCNPGBackupArtifact` and `marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added two test calls with the old argument lists. Both PRs were green on their own, and together they break `go test ./internal/backupcontroller/`. The two calls now pass the same default flavor as the other tests in the file. The second commit fixes an e2e suite that fails the required E2E check here. It is the `computeplane` suite from #4713, and its worker pool was one node with a single vCPU. After the kubelet reservations that node has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi, ingress-nginx, metrics-server, cert-manager) request exactly 900m between them. If cert-manager's startupapicheck hook pod is created last, it can't be scheduled and the install times out after 10m. The reinstall after remediation runs into the same full node. Whether that pod comes last depends on install order, so the suite passed on #4713 and failed here. The pool now has two workers with one vCPU each, the same shape the kubernetes suites use. One vCPU per worker stays on purpose, because of #3513. Two workers exposed a deadlock in the tenant cluster-autoscaler. The suite's pool is created at zero replicas and then gets a min size of 2, which CAPI does not apply to the live replica count on upgrade (#4786). The autoscaler's scale-up for a pending pod asks for one node, and the clusterapi provider refuses it because one is under the min size, although it is an increase. That refusal backs the group off, and the min-size enforcement skips groups in backoff, so the pool never leaves zero. Upstream had the same report in kubernetes/autoscaler#4942, closed as stale, and it is now tracked in kubernetes/autoscaler#10400. Our autoscaler image already patches the mirror case for max size, and the patch now refuses only a decrease below the min size. `go vet` and `go test ./internal/backupcontroller/...` pass, and so do the bats pins for the computeplane suite. The autoscaler patch adds a test for each direction, and the clusterapi package tests pass with it applied to 1.32.0. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cluster autoscaling for worker groups outside configured size limits: groups below their minimum can scale up toward it, and nodes can be removed when a group exceeds its maximum. Scale-down requests below the minimum remain restricted. * **Tests** * Updated backup and restore test calls with PostgreSQL and empty flavor values. * Updated ComputePlane checks for a two-worker pool and clarified cert-manager installation timing expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yan Bondarenko
|
dc362fa591 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent still takes its router IP and the Gateway API Ingress IP from its allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which the documented installs slice from the same range kube-ovn allocates pods from, so kube-ovn sooner or later hands a pod an address Cilium already holds. Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a /29 per node from the last /21 of networking.podCIDR, and kube-ovn excludes that /21 through --default-exclude-ips. The block stays inside the pod CIDR on purpose. Envoy reaches Gateway API backends from the node's Ingress IP, and kube-ovn masquerades host-originated traffic only when the source lies in one of its subnets, so a block outside them such as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes. A post-upgrade hook moves existing clusters. It first checks that the block has a /29 for every node and lies inside kube-ovn's default subnet, and stops without changing anything otherwise. It adds the block and every node's current Ingress IP to excludeIps of that subnet, because the agent keeps a former Ingress IP in its ipcache as reserved:ingress. It then waits until kube-ovn stops offering the block and stops if an address in it is still allocated. kube-ovn leaves excluded addresses out of the subnet status, so that check reads its IP objects. Then, one node at a time, it clears the inherited /24 from the CiliumNode, waits for the operator to allocate from the block and restarts the agent. networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn variants, and the render fails otherwise. A /21 cut into /29s also caps these variants at 256 nodes. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
4c3c838a66 |
test(e2e): run the computeplane pool at two workers
A single-vCPU worker has 900m to allocate after the kubelet reservations, and the tenant addons' CPU requests use all of it. When cert-manager's startupapicheck hook pod is the last one created it stays Pending, the install times out, and the reinstall after remediation runs into the same full node. Whether it is last depends on install ordering, so the suite passed or failed on the race. Two workers match the kubernetes suites' pool and leave room for the hook pod. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
bcda5c893a |
feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves the platform to declarative volume hotplug. It changes how every volume hotplug on the platform works, so the title carries `!`. Closes #4032. This PR is stacked on #4451 and has to merge after it. #4451 adds the kubevirt-csi flag this PR flips. The replacement gate is already on in KubeVirt 1.9, and the old gate is the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at v1.9.0, and Beta gates are on by default. virt-controller skips `handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so dropping the old gate is what switches the new one on. The kubevirt-csi fallback goes off in the same change. With the old gate off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments next to the gate and the flag point at each other. This needs the rebuilt kubevirt-csi image. A driver built before #4451 does not know the flag and exits on it at startup. The image is stamped at release preparation and ships in the same release as this change, so only a `make apply` from main before then would crash-loop the CSI controller. What changes at runtime on an upgrade: - The VM-level `addvolume` and `removevolume` calls now edit `VM.spec` directly. virt-controller then makes the VMI hotplug volumes match it, instead of going through `VM.status.volumeRequests`. - Volumes hot-plugged through the VM subresource are in `VM.spec`, so they stay attached. - During a tenant teardown, when the VM goes before its VMI, the driver no longer calls the VMI subresource. It waits for the VMI status to drop the volume, and the garbage collector deletes the VMI with its VM. Upstream waits up to two minutes. If the VMI is still there after that, the unpublish fails and is retried. A new platform migration keeps old VMI-only volumes attached. On its first pass virt-controller removes every hotpluggable VMI volume that is missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at the VMI level (the Role then granted `virtualmachineinstances/addvolume`), and live migration carries such a VMI across KubeVirt updates. Without help, a tenant node not recreated since then would lose a mounted disk on upgrade. Migration 59 copies each such volume and its disk from the VMI into the VM template before anything else changes. Adding a hotplug volume to the VM spec is a valid live update in KubeVirt, so no VM needs a restart. It runs as the platform chart's pre-upgrade hook, before the upgrade re-points `cozystack-packages`. The KubeVirt CR and the tenant kubernetes charts both render from that source, so the gate and the flag cannot change before the migration has finished. It only touches VMs that need it, logs each one, and a second run changes nothing. The migration also makes ephemeral hotplugs permanent. A volume a user added at the VMI level, for example with `virtctl addvolume` without `--persist` from an older virtctl, has the same shape as kubevirt-csi's old attachments, because virt-api builds both the same way. There is nothing reliable to tell them apart by, so both are copied. Leaving the user's volume out would not keep it ephemeral: the first reconcile would unplug it from the running VM. After the upgrade it is part of `VM.spec` and attaches again on a restart, and `virtctl removevolume` drops it. Ephemeral hotplug stops working on VMs. With the old gate off, virt-api refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns (`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and removed through the VM from then on. A VMI without a VM is not affected. Only running VMIs are copied. virt-controller reconciles no other VMI, and a stopped VM builds its next VMI from `VM.spec`, so copying from a stopped one would bring back volumes the restart was going to drop. Migrations run only on the isp-full, isp-full-generic and isp-hosted variants. KubeVirt comes from the platform only on isp-full and isp-full-generic, so those are covered. On the default variant, where packages are installed by hand, nothing runs the migration. This optional check lists the VMIs and volumes the migration would copy. It uses the same rule. On the default variant, run it before upgrading and recreate each tenant node it lists: ```bash kubectl get vm,vmi --all-namespaces --output json | jq --raw-output ' ([.items[] | select(.kind == "VirtualMachine") | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}] | from_entries) as $vm | .items[] | select(.kind == "VirtualMachineInstance") | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner | "\(.metadata.namespace)/\($owner)" as $key | select(.status.phase == "Running" and $owner != null and $vm[$key] != null) | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true) | .name as $n | select($vm[$key] | index($n) | not) | "\($key) \($n)"' ``` Empty output means nothing is affected. I checked the query against the migration's test fixtures only. Neither the query nor the migration has run on a live cluster. #4287 also adds a migration 59. If it merges first, this one moves to 60: the file, the version it stamps, `migrations.targetVersion`, the bats file (its `run_migration` argument and expected `STAMP`) and `hack/testdata/migration-59-vmi-hotplug`. A VM with a pending hotplug request at upgrade time may also get stuck. virt-controller still processes `VM.status.volumeRequests` before the declarative handler, and it applies them through the VMI subresource, which virt-api rejects once the old gate is off. The sync then fails before the VM spec is updated, so nothing clears the request, and every later sync of that VM fails the same way. A retry through the VM subresource edits the spec and clears it, which covers kubevirt-csi's own requests. A one-shot request stays stuck until the VM restarts. I read this from the v1.9.0 code and haven't reproduced it. Waiting until this lists nothing before upgrading narrows the window, but a request made during the rollout can still hit it: ```bash kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name' ``` The two halves live in different HelmReleases, so there is a short window during the rollout where the driver and the gate disagree. In that window an unpublish fails and is retried until both are updated. A tenant cluster whose `kubernetes` HelmRelease is suspended or failing keeps its old driver for longer. Drivers from before July 2025 attach through the VMI subresource, so on such a tenant every attach is refused until its release reconciles. Such a driver can also attach a volume after the migration has run and before the gate flips. The migration doesn't see that volume, and the first declarative reconcile unplugs it from the running VM. The migration lists the VMs once and patches each one with a `resourceVersion` test, so a VM that changes in between fails the hook Job. The Job has `backoffLimit: 3`, and each retry lists again and skips the VMs already patched. Re-reading only the conflicting VM would avoid the retry; I left that out. The new gate is listed in the CR even though it is on by default. With the old gate gone it is the only hotplug path, and the chart refuses a render that disables it. I found no other user of `HotplugVolumes` in the platform or in the downstream repositories. The VMI-level fallback may have been failing already. The kcsi Role has no rule for the VMI-level `removevolume` (#4452), so that call likely got 403 with or without the gate. I haven't checked that on a cluster. `hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left alone (a VMI that is not Running among them), a second run, a cluster without KubeVirt and the fail-closed paths. The helm-unittests for the KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in `disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The CSI test pins the flag at false, and the `make update` idempotency test still passes. None of that exercises hotplug itself. The real test is e2e with volume attach and detach on a tenant cluster, including a teardown. No CI lane upgrades from the previous release yet, so an upgrade with a live VMI-only volume is untested; #4463 tracks that test. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Declarative hotplug volumes are enabled by default for KubeVirt. * Running virtual machines keep eligible hotplugged volumes and disks when their configuration is migrated. * **Bug Fixes** * The migration avoids adding duplicate volumes or disks and does not mark the migration complete if required operations fail. * The Kubernetes CSI driver no longer uses the VMI hotplug fallback. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
d367846030 |
test(e2e): cover the computeplane module end to end (#4713)
## What this PR does This adds an e2e suite for the ComputePlane tenant module, which had only render tests so far. The suite lives in `hack/e2e-chainsaw/computeplane/`. It turns the module on in its own tenant, `tenant-cplane`, so the shared `tenant-test` is never toggled. That tenant is a child of `tenant-root`, not of `tenant-test`, because a tenant two levels down cannot reach the etcd it inherits from root (#4726). What it checks on a real cluster: - the module release installs, and the cluster release it renders keeps the pinned addons and the `internal.cozystack.io/computeplane` label; - the pool release gets only the keys from the module's per-pool allowlist (the patch sends a `nameOverride` through the ComputePlane API and checks it is dropped); - Kamaji writes `computeplane-cluster-admin-kubeconfig`, and the tenant can't get at it: the Secret is not labelled `tenantresource=true`, `tenantsecrets` does not list it to the tenant's super-admin group, and `kubectl auth can-i get` answers `no` for that group (the read was possible until #4722); - cert-manager and ingress-nginx come up inside the cluster, and the Proxied ingress Service gets a ready endpoint; - the module and cluster releases have no remediation cycle in `status.history`; - on teardown, the in-cluster addon releases are gone before the KamajiControlPlane starts deleting; - turning the module off removes all of it: releases, in-cluster addons, control plane, Secret, worker VMs and disks, and the LINSTOR pools go back to their pre-suite level. The issue names the Secret `computeplane-admin-kubeconfig`. The module writes `computeplane-cluster-admin-kubeconfig`, so the suite checks that name. The issue also lists fluxcd as a pinned addon. That is out of date. The module runs no in-cluster Flux on purpose (`packages/extra/computeplane/templates/cluster.yaml:108-110`), so the suite checks cert-manager and ingress-nginx only. One part of the issue is not covered here. The `placement: ComputePlane` routing in cozystack-api doesn't exist yet, so there is no catalog app to run on the cluster. The issue ties that to "together with, or right after" this test, so I left it for the routing work, tracked in #4715. Things the review found outside this diff, each filed on its own: the module release's 10m install timeout against a slow cluster bringup (#4714), the module docs not matching the per-pool allowlist (#4716), and the LINSTOR helpers in `_lib` running `kubectl exec` without a timeout (#4111). The suite brings its own pool: `md0`, one worker, same sizing as the kubernetes suites. The default `md0` that the module renders for empty `nodeGroups` asks for the `replicated` StorageClass. The container lane doesn't have it, so on this lane the default pool stays covered by the render tests only. ComputePlane workers can't use the in-sandbox Talos image cache. The pool allowlist drops `talos`, so the worker pulls its disk from the public Image Factory (#4700). If the worker join or an addon release times out, the catch block prints the worker image source and points at that issue, so a red run caused by the factory is easy to tell apart. The suite itself needs no entry in `hack/select-e2e.sh`, because `computeplane-application` maps to it by name. That source depends on `kubernetes-application` and re-sources the kubernetes charts. So PRs that touch the kubernetes charts or ingress-nginx now select `computeplane` too, which I estimate adds 12-15 minutes to a green run (kubernetes-previous takes about 8 minutes with two workers, plus teardown). A red run can take much longer, because the waits sit above the release install timeouts. The suite header has the numbers. It is the third suite that brings up a tenant cluster, and the 215m E2E job cap is sized for the red path of the kubernetes pair only. If worker join breaks everywhere, all three go red and can run past the cap, which cancels the job and loses its log. I left that as an accepted risk. `hack/select-e2e_test.bats` now expects `computeplane` in the ingress-nginx selection. The computeplane source also owns `apps/kubernetes-nodes`. Before this PR a pool chart edit reached no suite and ran everything. With the new suite alone it would have selected only `computeplane`, which never creates `KubernetesNodes` natively. So `kubernetes-nodes-application` now maps to `kubernetes-latest kubernetes-previous computeplane` in `hack/select-e2e.sh`, and two new tests pin that for a pool chart edit and a `kubernetes-nodes-rd` edit. The suite is not in the leak-guard allowlist. Its teardown already fails when a TenantControlPlane sticks on its finalizer (#4489), because the admin kubeconfig Secret it owns stays behind, so an exemption would only hide other leaks. The README suite list now names the suite. JFYI, not a defect claim: the Proxied ingress Service from the kubernetes chart has `internal.cozystack.io/tenantresource: "true"`. So the tenant sees it, even though `computeplane-rd` has `services.include: []`. The module doesn't promise to hide it. How I checked it. `chainsaw lint` passes at v0.2.15, the version the sandbox pins. I also ran the suite against an envtest apiserver, with fake controllers applying the real `helm template` output of the tenant and computeplane charts. The intact run is green. Each of these breakages turns it red at the step meant to catch it: label removed, addon `exposeMethod` changed, a key outside the pool allowlist, the allowlist `pick` replaced by the whole group, stale Ready from an older generation, wrong Secret name, Secret labelled `tenantresource`, Secret listed in `tenantsecrets`, a direct RBAC grant on the Secret, ingress-nginx addon not Ready, no ready ingress endpoint, an `uninstalled` history entry, the control plane deleted before the addon releases, an addon release or worker VM left after teardown. Closes #3369 Closes #4747 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): add an end-to-end suite for the ComputePlane tenant module ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added end-to-end coverage for enabling and disabling the ComputePlane tenant module, including generated releases, pool value filtering, admin credential access, addon and cluster readiness, and resource cleanup. * Kubernetes nodes changes now select the ComputePlane suite alongside the latest and previous Kubernetes suites. * **Documentation** * Added the ComputePlane suite to the documented end-to-end test suite list. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
80eb2e5037 |
ci(e2e): refuse a new fork image before the build starts
The fork e2e lane pushes a fork's image archives only under names the base tree already exports, which keeps a fork from minting repositories under the CI registry. An image the PR itself adds is therefore refused, but only at the push, after the whole build has run, and nothing the contributor does on the PR can clear it. Compare the image names of the PR tree and its base in the plan job of fork runs, with the same make -n derivation the push allowlist uses, and fail there naming the new images and what has to happen first. The push allowlist is unchanged and remains the boundary: this check runs from the fork's own tree, so it only saves an honest contributor the wasted build. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
9734a16692 |
feat(postgres): add a postgis image flavor (#4587)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Adds a `flavor: postgresql | postgis` enum to the `postgres` app. `postgis` runs the cluster on the CloudNativePG PostGIS operand image, so tenants can enable `postgis`, `postgis_raster`, `postgis_topology`, `postgis_sfcgal`, `postgis_tiger_geocoder` and `address_standardizer` through the existing `databases.<name>.extensions`. The default stays `postgresql` and renders exactly the image it rendered before this change. This picks up #2672 by @matthieu-robin, which the stale bot closed after a rebase request, with these changes: - The postgis tags in #2672 (`17.7-3.5`, `16.11-3.5`, ...) do not exist in `ghcr.io/cloudnative-pg/postgis`; the registry answers 404, so the flavor would have ended in `ImagePullBackOff`. The map now uses tags that exist, `<minor>-<postgis>-standard-trixie`, on the same PostgreSQL minor as `files/versions.yaml`. - v18 is supported: PostGIS images for 18 now exist upstream. - `standard` rather than `system`: the chart archives through the barman-cloud plugin since the backup rework, so the operand no longer needs the barman binaries of the deprecated `system` variant. - `hack/update-versions.sh` derives the postgis map from the minors it picks, and `hack/check-postgres-postgis-versions.bats` fails if the two maps drift to different minors or away from `standard-trixie`. - The chart refuses a flavor change on a live `Cluster`. The default images are built on Debian bullseye (glibc 2.31) and the postgis ones on trixie (glibc 2.41): CNPG would roll the swap out as a plain image update, under an initialised data directory, which silently invalidates every index on collatable text, and the postgis-to-postgresql direction also drops the libraries behind every PostGIS object. Only the image name is compared, so minor bumps and registry mirrors keep passing. - `pgrouting` is not claimed: it is absent from the `18.1-3.6.1` build the chart pins and only appears in later postgis builds. A general mechanism (CNPG ImageVolume extensions) was considered and left out: it needs PostgreSQL 18 and Kubernetes 1.35 or the `ImageVolume` gate, so it cannot serve v13-v17 nor most management clusters today, and PostGIS is the only extension CloudNativePG ships as a full operand image. No other extension request exists in this repository. Testing: - `helm unittest packages/apps/postgres`: 145/145, including 6 new cases in `tests/flavor_test.yaml` (default image, postgis on the Cluster and on the init Job, the guard in both directions with a mocked live Cluster, and a mirrored digest-pinned image taking a new minor). Removing the guard makes exactly the two guard cases fail. - `hack/cozytest.sh hack/check-postgres-postgis-versions.bats` passes, and fails on a shifted minor or a `system` tag. - `make -C packages/apps/postgres generate` leaves no diff. - The rendered `ghcr.io/cloudnative-pg/postgis:18.1-3.6.1-standard-trixie` was started locally: `CREATE EXTENSION` for citext, postgis, postgis_raster and postgis_topology succeeds and `ST_Buffer` answers. A three-instance cluster on the `system` sibling of that image has been running PostGIS in production since 2026-09-29. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: cozystack/terraform-provider-cozystack#63 (issue: the provider models every field by hand, and the default handling needs a decision there) - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(postgres): add a `flavor` field accepting `postgresql` (default) or `postgis`. `postgis` bootstraps the cluster from the CloudNativePG PostGIS image, so PostGIS extensions can be enabled per database through `databases.<name>.extensions`. The flavor is fixed at creation: changing it on an existing cluster is refused, and so is restoring a backup into a cluster of the other flavor, whether through a RestoreJob or `bootstrap.enabled`; move data across flavors with a logical dump. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added a `flavor` option for PostgreSQL clusters. The default `postgresql` flavor remains available, while `postgis` selects an image with PostGIS extensions. - PostGIS images are available for PostgreSQL versions 13–18. - Cluster flavor is fixed at creation. To switch flavors, create a new cluster and migrate data with a logical dump. - **Bug Fixes** - Restores are refused when the backup and target cluster use different flavors. - **Documentation** - Updated PostgreSQL configuration guidance to describe flavor options and limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
cc51cac63d |
fix(monitoring): give a central tenant's Grafana a login that cannot write
The collector holds a central tenant to its write rate, but the one login it writes with was also the login of the tenant's Grafana datasource, and it could both read and write. vmauth cleans the request path before it picks a route, so any user the tenant admits to its Grafana, a Viewer included, could POST through the datasource proxy to ../insert and write into the shared store past the limiter. Each central tenant now has two VMUsers on its one account: the collector's, which may only write, and Grafana's, which may only read, each with its own chart-owned Secret and password. A namespace name holds no dot, so the reader's username never matches another tenant's. On upgrade the datasource moves to the new login in the same release that creates it, so reads may fail until vmauth loads the new user. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
474377e8ca |
build(monitoring): build the traces collector with a rate limiter
A shared-central traces store needs each tenant held to a write rate, or one tenant can fill the disk every tenant shares. No upstream OpenTelemetry distribution or processor limits throughput. Elastic's EDOT Collector registers Elastic's ratelimitprocessor among its default components, but it ships under the Elastic License 2.0 and carries far more components than the chart uses. The collector is therefore built here with the OpenTelemetry Collector Builder from the upstream components the chart already uses, at the same v0.136.0, plus that ratelimitprocessor (Apache-2.0). Its v0.8.2 is the release built against v0.135.0, the last before v0.137.0. The image is versioned by the collector release with a cozystack respin counter, runs as the upstream image's user, and is built with the other package images. The builder stage is alpine, which has no git, so a proxy error during its go install would have had neither the direct fallback nor a retry; the stage installs git. The module-fetch contract saw only go build, so it now checks go install too, and takes only git as cover for a versioned go install, which resolves outside the stage's module. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
d35493a64d |
test(e2e): cover the computeplane module end to end
The module had render tests only. Nothing checked that turning it on in a tenant yields a working cluster: the Kamaji control plane, a worker pool shaped by the module's allowlist, an admin kubeconfig the tenant can neither list nor read, the pinned cert-manager and ingress-nginx addons running inside the cluster, and a teardown that removes the in-cluster addons before the control plane and leaves nothing behind. The suite runs the module in its own child tenant so the shared tenant-test is never toggled. It supplies its own pool because the default md0 asks for the replicated StorageClass, which the container lane does not have. On failure it prints the worker image source, since the pool allowlist drops talos and a ComputePlane worker cannot use the in-sandbox image cache. The computeplane source depends on the kubernetes application and runs its own ingress-nginx, so an ingress-nginx change now selects this suite too. It also owns apps/kubernetes-nodes, which used to reach no suite and run everything; kubernetes-nodes-application now maps to both kubernetes suites and computeplane, so a pool chart change keeps the suites that create KubernetesNodes natively. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM |
||
|
myasnikovdaniil
|
a5d12e78d4 |
fix(ci): reconcile three unit-test contracts broken by adjacent merges (#4751)
## What this PR does Unit tests on main are red, and that blocks E2E on every open PR. Three contracts broke the same way. In each case a check from one PR and a line from another merged a few minutes apart on 2026-10-05, with no textual conflict. Any one of them alone keeps the job red, so they are fixed together. The first is the pin on how a suite reads HelmRelease history. #4689 added it. It accepts `$(kubectl ...` and, since #3594, the bounded `$(_cozy_guard_kubectl ...` wrapper. Nothing in front of kubectl is allowed. The monitoring suite from #4678 bounds that read with `$(timeout 30 kubectl ...`, and the bound is wanted. So the pin now accepts an optional `timeout [-k N] N` in front of kubectl. The prefix is spelled out on purpose. Any text would also let through `$(echo kubectl ...)`, which names the shared expansion without running it, and that form is now a refuse case. I checked it with mutations: the new accept cases fail against the old pattern, and a pin that takes any prefix fails a refuse case. The second is the flux install in workflows. #4645 forbids running fluxcd.io/install.sh and moved every workflow to the SHA-pinned fluxcd/flux2 action. The isp-hosted lane from #4712 still ran the script. It now uses the same action and version. The third is the negated-assertion audit. It reads the find operand in `\( ! -type d` in select-e2e_test.bats as a shell negation and fails. The line now says `-not -type d`. This is the same change as #4748, so this PR supersedes it. The audit bug itself is #4752. After the Go fix, the full log of the unit job had only these two bats failures left. Closes #4750 Closes #4755 ### Screenshots ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): fix three unit-test contracts that went red on main after adjacent merges. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * History checks now recognize direct `kubectl` reads and reads through the supported wrapper, including timeout-bounded commands with an optional kill-grace setting. Commands that only include `kubectl` as an argument, use similarly named commands, or place a timeout before the wrapper are not treated as reads. * **Chores** * The hosted end-to-end workflow now uses a pinned Flux version, and suite selection handles directory exclusions consistently. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0cae0e87ad |
test(e2e): spell the find negation as -not in the suite-list helper
The negated-assertion audit splits a line on `(` and reads the `!` after `\(` as a shell negation that errexit would exempt, so the helper's find predicate fails the audit and the unit tests on main. `-not` is the same predicate in GNU, BSD and busybox find and does not look like one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7bf6906a80 |
fix(platform): keep the legacy etcd-operator stopped after etcd adoption (#4738)
## What this PR does Migration 50 scales the legacy etcd-operator to 0, adopts every legacy `etcd.aenix.io` cluster with `etcd-migrate`, and then scales the operator back to 1. At that point the Deployment still runs the legacy operator, and after a successful adoption it has nothing left to manage. Bringing it back undoes part of the adoption. Observed on a production cluster upgraded from 1.4.3 to 1.6.4. The API audit log and the Helm history of a tenant's `etcd` release show this sequence, in UTC on the same day. - 13:21:12 migration 50 scales `cozy-etcd-operator/etcd-operator-controller-manager` to 0. - 13:22:59 `etcd-migrate` updates `etcd-headless`, pointing its owner references at the adopted EtcdMembers, and orphan-deletes the legacy EtcdCluster and StatefulSet. - 13:23:04 migration 50 scales the operator back to 1. It is still the legacy operator, the v1alpha2 one replaced it only at 13:40. - 13:23:42 Flux upgrades the tenant's `etcd` release with the old chart (Helm revision 87), because the new chart artifact appeared only at 13:24:40. The old chart re-creates the legacy EtcdCluster at 13:23:43. - 13:23:43 to 13:23:45 the legacy operator reconciles it, updates `etcd-headless` twice and re-creates the legacy StatefulSet `etcd`. - 13:36:49 Flux upgrades the release to the new chart (revision 88), which prunes the legacy EtcdCluster. - 13:36:50 the garbage collector deletes `etcd-headless` and the re-created StatefulSet. The same happened in all five tenants that run an etcd. The pods survived because EtcdMembers own them, but they keep `spec.subdomain: etcd-headless`, so their `etcd-<i>.etcd-headless.<ns>.svc` names stopped resolving. From then on every member logged `no such host` for its peers, the v1alpha2 EtcdCluster had no status conditions, and the `etcd-defrag` CronJob failed every hour with `context deadline exceeded`. The transitional `etcd-headless` Service the etcd chart ships for this window (#3270) does not cover it. Helm patched the existing Service 202 ms before the garbage collector removed it, and did not create it again because it was already part of the release. With this change the operator stays at 0 after a successful adoption. The etcd-operator HelmRelease replaces the Deployment with the v1alpha2 operator later in the same upgrade, and no later migration waits for the operator. A failed or interrupted adoption still restores it to 1 through the existing EXIT trap, so a failed run does not leave the tenant etcds without an operator. If that HelmRelease cannot reconcile, the adopted clusters wait for it with or without this change, because the legacy operator does not reconcile v1alpha2 EtcdClusters. Clusters already upgraded with this bug can be repaired without restarting anything. Creating the Service from the tenant's own `etcd` release is enough. ``` helm -n <ns> get manifest etcd | awk 'BEGIN{RS="\n---\n"} /\nkind: Service\n/ && /\n name: etcd-headless\n/' | kubectl -n <ns> create -f - ``` On the cluster above the v1alpha2 EtcdCluster reported `Available` with reason `QuorumHealthy` within 11 seconds of the Service being created, in every tenant, and the members stopped logging `no such host`. ### Validation - `hack/cozytest.sh hack/migration-50-etcd-adopt.bats` passes all 29 tests. A new test asserts that a successful adoption scales the operator to 0 and never back to 1. Against the current script it fails with `FAIL: the legacy operator was scaled back up after a successful adoption`. - The existing test for a failed adoption still passes, so the EXIT trap still restores the operator to 1. - The new test checks the absence with `if grep ...; then echo "FAIL: ..."; false; fi`, the form the other suites use. A bare `! grep` assertion cannot fail a test under `hack/cozytest.sh`, so it could not have caught this. The other negated assertions in this file had the same weakness, and #3790 has rewritten them on main. With all of them made able to fail in a scratch copy, the suite still passes against this change. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. It changes a migration script and its bats test. No package, values, schema, CRD, make target or `hack/` layout changes, so no entry matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): keep the legacy etcd-operator stopped after migration 50 adopts the legacy etcd clusters. Scaling it back up let it reconcile the legacy EtcdCluster that Flux re-created from a not yet upgraded etcd chart, re-create the legacy StatefulSet and take the legacy etcd-headless Service away from the adopted members, which the garbage collector then deleted. The adopted members lost their DNS names, the EtcdCluster never reported a status and etcd defrag failed every hour. Clusters already upgraded can recreate the Service from the etcd release manifest. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Migration Updates** * After successful etcd adoption, the legacy operator remains scaled to zero replicas for replacement by the etcd operator; it is no longer scaled back up before the migration is stamped. * If adoption fails or exits before completion, the legacy operator is restored to one replica. * The dry-run message now describes the zero-replica outcome of successful adoption. * Successful adoption continues to record the migration version stamp. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
415430bcc7 |
fix(e2e): fail on a failed release of cilium and csi as on the parent, and report addon remediation (#3594)
## What this PR does The history guard in the tenant Kubernetes e2e suite checked only the parent `kubernetes-<test>` HelmRelease. A failed action on one of the addon releases it installs left nothing the suite looked at, so a run showed only the downstream symptom, usually the node-join deadline, and never named the release behind it. The guard now reads every addon release of the parent. It selects them from the cluster by name prefix, so a release added to the chart later is covered too. The rule: - A `failed` or `uninstalled` Snapshot fails the run on any release whose install strategy is `RetryOnFailure`. That is the parent (the aggregated apiserver stamps it on every Application HelmRelease) and the tenant `cilium` and `csi`. - The same footprint on the other addons, which run the default strategy with `retries: -1`, is printed as a note and the run stays green. For the CRD addons that configuration is tracked in #3555. A transient `cilium` or `csi` failure that later recovers on a retry still turns the run red, by design, as long as its Snapshot still reads `failed`. One recovery leaves no such Snapshot: when an install fails and the very next attempt succeeds, Helm marks the failed install `superseded`, because there is no deployed release to supersede, and the history reads like an ordinary upgrade. The guard cannot see that case from the statuses alone. The code and the docs now say so, a test pins it, and detection is left to a follow-up. An earlier revision of this PR tried to tell a teardown apart from a retried failure and made only `uninstalled` fatal. That split does not hold. In helm-controller v1.5.0, the version the platform installs, a release with an active retry strategy never remediates. `atomic_release.go` returns an upgrade for a failed release before it reaches the remediation switch, and the two other paths that uninstall (a changed release target, and deletion) both clear history afterwards. So on these releases `uninstalled` never shows up, and `failed` is the only signal. The parent keeps the rule it already had on main, and `cilium` and `csi` now follow it too. The failure paths get the same reading. The guard runs after the assertions, while the deadlines a failed addon breaks end the script earlier, so the same classification is also printed, without a verdict, from the node-join collector and from the exit handler. That handler is now armed at the tenant CR apply, so the management-cluster waits before the tenant answers are covered too. Smaller changes that came with this: - Every guard read is bounded, and a failed read is never folded into "no history". - The parent must read back a non-empty strategy. An empty value there means the jsonpath stopped resolving, and without this check every footprint would quietly turn into a note. - The history read uses `HELMRELEASE_HISTORY_JSONPATH`, and the upstream-type pin in `internal/fluxcontract` now accepts the bounded wrapper the read goes through. `docs/agents/e2e-testing.md` §6 and checklist item 8 describe the new rule. On related issues: this adds the `local` that #3774 reports missing in `helmrelease_has_remediation_cycle`, but not the sweep over the other `_lib/` helpers that the issue also asks for. For #3773 it rewrites the guard comment the issue quotes. The waits without a backstop are not touched. Closes #3553. ### Screenshots Not a UI change. ### Downstream repositories I walked the trigger map against the diff. It touches `hack/e2e-chainsaw/_lib/`, `hack/*.bats`, one Go test under `internal/fluxcontract`, two spend-order comments in the kubernetes suites, and `docs/agents/e2e-testing.md`. No package, schema, CRD, make target or user-facing document changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): the HelmRelease history guard now also covers the tenant addon releases. A failed or uninstalled Snapshot fails the run on the parent, cilium and csi, which retry in place; other addons get a note. The same classification is printed on the failure paths where the guard itself cannot run. ``` |
||
|
IvanHunters
|
2188f16461 |
test(vminstance): cover VMDisk clones from an image and from a disk (#4586)
## What this PR does Closes #3482. The `vminstance` suite now covers both VMDisk clone paths, which had no e2e test at all. One disk is cloned from a golden image with `source.image`, and a second disk is cloned from the first with `source.disk`. For each clone the test checks that the DataVolume points at the expected source PVC and becomes `Ready`, that its PVC is `Bound`, and that the VMDisk reports `Ready` and `WorkloadsReady`, the same way the standalone VMDisk test does. The `Bound` check is separate because CDI derives `Ready` from the DataVolume phase alone, and can reach `Succeeded` from the claim's annotations before the claim binds; a `phase: Succeeded` assert would only repeat `Ready`. The golden image is a fixture the suite creates itself. `vm-default-images` is an opt-in package and the e2e install does not enable it, so `alpine-3.21` from the default collection is not there to clone. The suite imports the same Alpine 3.21 image into `cozy-public` under the name `e2e-alpine-3.21`, so it can never collide with an image the package owns. This is the only download in the new test, and the `source.disk` case reuses the first clone as its base disk. The long wait sits on the DataVolume, not on the HelmRelease. Since #4380 a vm-disk release goes Ready once the DataVolume exists and the clone's progress is reported through the VMDisk `WorkloadsReady` condition, so the test does not assert the HelmRelease at all. Chainsaw deletes the clones before the golden image at cleanup. The clone itself is only proven by this PR's E2E run, on the `local` class of the PR lane. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `hack/e2e-chainsaw/vminstance/`, and nothing in the trigger map covers it. ### Release note ```release-note test(vminstance): e2e now covers VMDisk clones from a golden image (source.image) and from another disk (source.disk). ``` |
||
|
IvanHunters
|
4b1a75a332 |
test(vminstance): dry-run evict the VM launcher pod (#4585)
## What this PR does Part of #4422. The e2e suite now asks KubeVirt whether it would let a VM launcher pod be evicted, without evicting it. Until now nothing in e2e evicted a launcher pod, so the drain behaviour set up in #4403 was known only from reading KubeVirt. Once the `vminstance` VM is running, the suite sends a dry-run Eviction (`dryRun=All`) for its virt-launcher pod and checks the answer. Nothing gets deleted. A dry run still goes through the KubeVirt eviction webhook, which is registered with `sideEffects: NoneOnDryRun` in v1.9.0. With `LiveMigrateIfPossible` its answer follows the VMI's `LiveMigratable` condition. That condition depends on storage, so the test reads it instead of assuming it: - `LiveMigratable=True`: the eviction must be refused with the webhook's `Eviction triggered evacuation of VMI "<ns>/<name>"` message (HTTP 429). A drain retries on that while the VM migrates away. - `LiveMigratable=False`: the eviction must be allowed. The old `LiveMigrate` value refused this case with "is not live-migratable". KubeVirt v1.9.0 creates no PodDisruptionBudget for a VMI, so no PDB can refuse it either. - Anything else, including a missing condition, fails the test. The allowed answer alone proves little, because the webhook also lets an eviction through under `None`. So the test also asserts `evictionStrategy: LiveMigrateIfPossible` in the VMI spec, where virt-api copies the cluster value at creation. After the dry run it checks that the launcher pod is the same object and not terminating, and that the VMI has no `evacuationNodeName`. On the PR lane only the `False` branch runs. Post-install prep pins `StorageProfile/local` to RWO because DRBD is off there, and an RWO disk makes the VMI non-migratable. Nightly and tag runs use `replicated`, where I expect the `True` branch, but I have not seen this suite run on that lane yet. A real drain and a workload update after a KubeVirt bump are still open in #4422. I ran the script against a stub `kubectl`. Both expected answers pass, and it fails on every wrong answer I tried, such as a refusal for a non-migratable VMI, a PDB-style 429, a missing condition, two launcher pods, or a replaced pod after the dry run. `chainsaw lint` (v0.2.15) and shellcheck pass. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `hack/e2e-chainsaw/vminstance/` and the suite README, and nothing in the trigger map covers either. ### Release note ```release-note test(vminstance): the e2e suite now dry-run evicts the VM launcher pod and checks the KubeVirt eviction decision against the VMI's LiveMigratable condition. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added end-to-end coverage for dry-run eviction of a running virtual machine’s launcher pod. The test checks the VM’s live-migration policy and `LiveMigratable` condition, expecting eviction to be refused when the VM is live-migratable and accepted otherwise. * Verified that the dry run leaves the launcher pod unchanged and does not record an evacuation node on the VM. * **Documentation** * Documented the test’s behavior and the local `jq` requirement for running it. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
50dec5d46a |
fix(platform): resume a retried migration Job from the stamped version (#4583)
## What this PR does A retried pod of the platform migration Job no longer runs migrations that already finished in the same upgrade. The Job gets `CURRENT_VERSION` once, at render time, and a failed pod is replaced by one with the same environment. By then every finished migration has stamped the `cozystack-version` ConfigMap, but the new pod ignored that and started from the rendered version again. `run-migrations.sh` now starts from the version stamped in the ConfigMap. If reading it fails, the pod fails and the Job retries it, because falling back to the rendered version is exactly the re-run this fixes. The rendered value is used only when the ConfigMap holds no integer version, which is also when the chart itself could not get a real version from it. Without that fallback a value like `v58` gave a green Job that ran nothing. The new `hack/run-migrations.bats` runs the script in the migrations image's alpine base against a fake kubectl that keeps the stamp in a file, so one pod's stamps are what the next pod reads. It needs docker, like the other migration suites. I also found that the ConfigMap existence check at the top of the script treats any API error as a missing ConfigMap and skips every pending migration. That is older than this change and is filed as #4579. Closes #4555 ### Screenshots ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): a retried platform migration Job now resumes from the version stamped in the cozystack-version ConfigMap instead of re-running migrations that already finished in the same upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Migration startup now resumes from a valid stamped version, avoiding reruns of migrations that have already completed. * Empty or non-numeric stamps fall back to the rendered version. Out-of-range stamps or failed stamp reads stop migrations before they run. * If the migration ConfigMap is missing, migrations do not run and the target version is stamped. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
2ddd55ce5b |
fix(build): honour PUSH and LOAD from the environment and in image-talos (#4654)
## What this PR does A local build that asks to load images into docker no longer pushes them. `hack/common-envs.mk` set `PUSH` and `LOAD` with plain `:=`, and a makefile assignment beats the environment, so `LOAD=1 PUSH=0 make image` quietly pushed a multi-arch build. Both now take the environment value, and an empty one falls back to the default. I also made `PUSH` default to 0 when `LOAD=1`, so `LOAD=1` alone only loads. Before this, `make image LOAD=1` pushed and loaded at once. An explicit `PUSH` from the environment or the command line still wins, and `OCI_EXPORT_DIR` still forces both off. Every buildx target gets the values through `BUILDX_ARGS`, so the fix lives in that one file. The Talos installer was the one image that ignored `PUSH`. It is published with skopeo, not buildx, so it never saw `--push=$(PUSH)`. Its registry copies now run only when `PUSH=1`. A build that pushes nothing no longer writes `PUSHED_TAGS_LOG`. That log tells the multi-arch stitch which tags to move, so it must hold only tags that were really pushed. The cache helper lost an argument nobody used. No caller passes the second argument of `cache-args`, and an explicit tag there would bypass `CACHE_TAG`, so the amd64 and arm64 builds of that image would share one cache ref. CI sets neither `PUSH` nor `LOAD`, so CI builds do the same thing as before. Each fix has a `make --dry-run` test in `hack/common-envs_test.bats`. Closes #4499 Closes #4504 Closes #4551 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The ccp skills pass `PUSH=1` explicitly, which still works. The website docs do not mention `PUSH`, `LOAD` or `cache-args`. ### Release note ```release-note fix(build): `LOAD` and `PUSH` set in the environment are now honoured, `LOAD=1` no longer pushes unless `PUSH=1` is also set, and the Talos installer is no longer published when `PUSH=0`. ``` |
||
|
Aleksei Sviridkin
|
2af8b0c229 |
ci(build): fall back to direct, retry and cache Go module downloads (#4660)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does One transient error from proxy.golang.org is enough to fail an image build or the `Unit & controller tests` job today. The error is an HTTP/2 stream reset (`stream error ... INTERNAL_ERROR`), and nothing retries or falls back after it. This PR adds three things: a direct fallback, a bounded retry and a module cache. Image builds now fall back to fetching modules from their origin when the proxy fails. They get `GOPROXY=https://proxy.golang.org|direct` as a build arg from `hack/common-envs.mk`, and every Go builder stage declares `ARG GOPROXY`. With a pipe Go goes direct on any proxy error. With Go's default comma it does that only on a 404 or 410. The value comes from `IMAGE_GOPROXY` and not from the host's `GOPROXY`, because buildkit can't reach a mirror on localhost or a `file://` proxy. Every workflow job that runs Go on the runner sets the same value as env. Explicit module downloads in Dockerfiles are retried three times with a growing pause. The loop is inline, in one form that the contract test pins. I tried a shared script from a named build context first, but buildx refuses any `--build-context` for the four Dockerfiles still pinned to the `dockerfile:1.2` frontend. The images also build from different contexts, so no single file can be copied in. Raising those pins changes how those images build and needs a real build to check, so I left it for a follow-up. Two alpine stages, token-proxy and shadowbox's `ss-server`, left the download to `go build`. The alpine golang images have no git, so the direct fallback can't fetch for them, and they now get the same download step. The runner jobs use `hack/go-mod-download.sh` with the same bound. The unit test job now starts from a module cache instead of downloading everything. It restores `GOMODCACHE` with `actions/cache`, keyed on every `go.sum` in the tree, and then prefetches all tracked modules through the retry, so a cache miss still works. A cache saved by a PR run is visible only to that PR, but one saved on the default branch can be restored by any PR ([docs](https://docs.github.com/en/actions/reference/workflows-and-actions/dependency-caching#restrictions-for-accessing-a-cache)). That is why the build cache warmer on main writes the same key before its image build. This way the first run of a PR, which is the one that flakes, starts warm. The warmer restores only an exact key, so old module versions don't pile up, and its Go steps can't fail the image build. Both runner labels are Linux, so the key matches. Both also check out under `/home/ubuntu/_work`, so `go env GOMODCACHE` should give the same cache path, but I have not seen it on the warmer's runner yet. The release jobs in `tags.yaml` and `promote-rc.yaml` get the GOPROXY env only, no cache. `hack/go-module-fetch-contract.bats` covers all three. Each test is red on main, and I checked each guard with a mutation. Closes #4591 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(build): Go module downloads in image builds and CI jobs now fall back to direct fetching and retry on transient proxy.golang.org errors. The unit test job reuses a module cache written from main. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Reliability** - Go module downloads now fall back to direct access when the public proxy encounters errors. - Builds and automated checks retry failed module downloads, helping them recover from temporary network issues. - **Performance** - CI caches Go modules to reduce repeated downloads and speed up builds and tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
9c025fa2b1 |
fix(installer): skip the packages artifact push and stamp with PUSH=0 (#4663)
## What this PR does A local build with `PUSH=0` no longer publishes the packages artifact. Before this change, `make image-packages` in `packages/core/installer` ran `flux push artifact` no matter what `PUSH` was set to, and then wrote the pushed digest into `values.yaml`. The `chart` target in the same Makefile already respects `PUSH`. With `PUSH` set to anything other than 1, the target now skips both the push and the digest stamp. It prints one line saying so and leaves the committed `platformSourceUrl` and `platformSourceRef` as they are. It still writes `packages/core/platform/.build-revision`, which is gitignored. I chose to skip the stamp instead of computing a local digest, because every reader of that value needs a digest that exists in the registry. The chart passes `platformSourceRef` to the operator as `--platform-source-ref`, and Flux then pulls `cozystack-packages@<digest>` from the registry. The release scripts `verify-promoted-packages.sh`, `promote-packages-artifact.sh`, `stitch-multiarch.sh` and `nightly-mirror.sh` also treat it as a registry digest. A digest of an artifact that was never pushed points at nothing. On main an unpushed run would not even get that far, since the recipe dies at `test -n "$DIGEST"`. The gate is a make-level `ifeq ($(PUSH),1)`, so it reads whatever `PUSH` ends up as after `hack/common-envs.mk`. On main that means `PUSH=0` on the command line. I also applied the `common-envs.mk` change from #4654 locally, which takes `PUSH` from the environment and makes `LOAD=1` default it to 0. With that applied, a dry run shows no push and no stamp for `PUSH=0` in the environment, for `LOAD=1`, and for `PUSH=0` on the command line, while `LOAD=1 PUSH=1` and an empty `PUSH` still push. CI is not affected, because no workflow sets `PUSH` or `LOAD` and `PUSH` stays at 1. I dry-ran the three CI invocations of this Makefile before and after the change, on main and again with #4654 applied: - `tags.yaml`, release stitch step: `make -C packages/core/installer image-packages chart` with `REGISTRY`, a release tag as `IMAGE_TAG`, and `PUSHED_TAGS_LOG` - `pull-requests.yaml`, installer step of a same-repo PR: `make -C packages/core/installer image` with a `pr-N-sha` tag, `PLATFORM=linux/amd64`, `BUILDER`, and an empty `OCI_EXPORT_DIR` - `build-main.yaml` and `build-release.yaml`: `make build`, which runs the installer `image` target with defaults ```text $ diff --recursive before/ after/ # tags.txt pr.txt build.txt $ echo $? 0 ``` There is one side effect. `OCI_EXPORT_DIR` forces `PUSH := 0`, so running `image-packages` with it set no longer pushes, where before it would have tried an anonymous push. No CI path does that. The fork PR build already runs only `pre-checks image-operator` for the installer, and the `make -n ... image OCI_EXPORT_DIR=/oci` scan in `e2e-fork.yaml` greps only the `.oci.tar` names, which this change does not touch. The new test dry-runs the target in both modes. With `PUSH=0` there must be no `flux push artifact` and no `platformSourceRef` stamp; with `PUSH` at its default, both must be present. The first test fails on main. I mutated the guard three ways, and each time only the matching test went red: - condition forced true: the `PUSH=0` test fails - compared against 2: the default-path test fails - `endif` moved up so only the push is gated: the `PUSH=0` test fails on the stamp check One limitation is left as a follow-up. The test's `env -u PUSH -u LOAD` only isolates it once #4654 lands, because on main a file assignment wins over the environment anyway. It does not cover `PUSH=0` coming in through `MAKEFLAGS`, so `make bats-unit-tests PUSH=0` would fail the default-path test here and the ordering check in `hack/stitch-multiarch_test.bats`. Nothing runs the suite that way today. If that changes, passing an explicit `PUSH=1` on the default-path `make` lines, or unsetting `MAKEFLAGS` the way `hack/bats-unit-keep-going.bats` does, would fix it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The website's `development.md` tells developers to run `make image-packages` with `REGISTRY` set to their own registry. That is the default `PUSH=1` path, and it works as before. Neither the website nor ccp mentions `PUSH=0` for this target. Closes #4656 ### Release note ```release-note fix(installer): `make image-packages` no longer pushes the cozystack-packages artifact or stamps its digest into values.yaml when PUSH=0. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * You can run `image-packages` with `PUSH=0` to skip publishing the artifact and keep the committed platform source reference unchanged. * When pushing is enabled, the command publishes the artifact and updates the platform source URL and digest-based reference. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
33cf6972bb |
test(qdrant): exercise the tenant trust anchor end to end (#4701)
## What this PR does The qdrant e2e suite now runs with TLS on and checks the tenant trust anchor end to end. Before this, the fixture left `tls` unset with `external: false`, so the chart rendered no CA and no `TenantProjection`, and nothing in e2e reached the controller or `tenantsecrets`. Setting `tls.enabled: true` turns TLS on without the LoadBalancer that `external: true` would allocate. The first new step checks that the anchor is published. The `qdrant-test-ca` sentinel has to report `Ready=True/Projected`, and `qdrant-test.tenant-ca` has to carry both the `tenant-ca` and `tenantresource=true` labels. The projection must hold only `ca.crt` with the same fingerprint as the chart's CA, and `tenantsecrets` must serve it. The key-bearing `qdrant-test-ca` and `qdrant-test-tls` must answer NotFound there, and any other error fails the step. The second step checks that the anchor is what decides trust. It reads the anchor through `tenantsecrets`, port-forwards the qdrant Service and runs `openssl s_client` with that file as the only CA, hostname check included. A CA with the same subject and another key must fail, and so must the runner's default trust store. A handshake failure counts only when the output shows a verification error, so a refused connection does not pass as a rejection. Plaintext HTTP must get no response. Curl exits 7 and 28 fail the step, and a second verified handshake afterwards proves the forward was still alive. TLS does not break the existing steps, because none of them talk to qdrant. `create-qdrant`, `wait-helmreleases-ready`, `verify-statefulset-and-storage` and `verify-pvc-reclaim-on-delete` only read Kubernetes objects: no `http://`, no port 6333 or 6334, no curl or exec, and the suite has no backup round trip. The release now also renders two Issuers, two Certificates and the sentinel. The chart switches its probes to HTTPS with `enable_tls`, so `readyReplicas: 1` now covers a pod serving TLS. Local checks: `chainsaw lint test` (v0.2.15) says the file is valid. `helm template` with the fixture's spec renders 2 Certificates, 2 Issuers and 1 TenantProjection, and none without TLS. I ran the handshake logic against a local `openssl s_server` with a CA and leaf shaped like the chart's. The anchor passes, the wrong CA and the default store fail on verification, a wrong hostname fails, plaintext gets `000`, and a closed port is not counted as a rejection. A sentinel with a missing source (`SourceNotFound`) is not tested here, the reconciler unit tests cover it. Review notes I left as they are: - Through `kubectl port-forward` the local socket is always accepted, so curl exits 7 and 28 only catch a dead forward process. A false pass would need qdrant to answer plaintext and a pod-side blip on that one request at the same time. - qdrant may reset the socket on the plaintext request, and some kubectl versions drop the whole port-forward on a pod-side error. Then the handshake after it turns the step red, not green. If CI shows `lost connection to pod` there, the plaintext check should get its own port-forward. - The suite's `catch` has no `describe` of the `TenantProjection`. The sentinel assert prints its conditions when it fails, but a red Secret assert or script after it would not. - The vendored chart's `servicemonitor.yaml` hardcodes `scheme: http`, so metrics scraping fails for any qdrant with TLS on. That was already the case before this PR, and the suite does not check metrics. Tracked in #4703. cert-manager does not set owner references here, so `qdrant-test-ca` and `qdrant-test-tls` stay in the namespace with their private keys after the app is deleted. That is tracked in #4704. The nats half of #3706 is tracked in #4702. Closes #3706 ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `hack/e2e-chainsaw/qdrant/`, which no entry in the trigger map covers. ### Release note ```release-note test(qdrant): e2e now runs qdrant with TLS on and checks that a client trusting only the tenant-published CA reaches it, while a wrong or missing anchor and plaintext HTTP are refused. ``` |