mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
docs/api-stability-policy
7639
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tym83
|
8fe7fd1aa1 |
docs: publish API stability and deprecation policy
Adopters interviewed during the CNCF Incubation due diligence asked for a written contract on API stability, and the roadmap committed to one for 2026 Q3. The policy defines alpha/beta/stable levels, deprecation windows, supported upgrade paths and the rule that breaking changes to existing objects always ship with a migration. The GTR and the roadmap now point to it instead of listing it as outstanding. Assisted-by: LLM Signed-off-by: tym83 <6355522@gmail.com> |
||
|
myasnikovdaniil
|
55aad6f914 |
fix(tcp-balancer): cap HAProxy fds and threads by the pod's own limits (#5005)
## What this PR does
On a v1.6.5 cluster a TCPBalancer created with the chart defaults
(`t1.nano`, limits 250m CPU and 128Mi memory, `haproxy:3.4.4`)
crash-looped. Both pods were OOMKilled with a memory limit of 134217728,
7 restarts each in about 15 minutes, and HAProxy logged `[NOTICE] (8) :
Automatically setting global.maxconn to 524126.` On another cluster the
same chart picked maxconn 524251 and the pods kept running.
`haproxy.cfg` has no `global` section, so HAProxy sizes itself from the
node. maxconn comes from the nofile limit capped at the built-in
`fd-hard-limit` of 1048576 (`DEFAULT_MAXFD`), and the thread count is
the number of CPUs the process may run on, whatever its CPU limit is.
Both are paid for at startup, before any traffic, and the thread part
grows with the node.
This PR adds a `global` section sized from the replica's own limits,
read through the same `cozy-lib.resources.defaultingSanitize` helper the
Deployment uses, so `resources` and `resourcesPreset` resolve the same
way. For `t1.nano` it renders
```
global
fd-hard-limit 4096
thread-hard-limit 1
```
`fd-hard-limit` is 32 per MiB of the memory limit and HAProxy derives
maxconn from it, 2035 for `t1.nano`, which is about 64 KiB per
connection. For comparison `haproxy -m 128` picks maxconn 3600 on 3.4.4,
about 35 KiB per connection, and the difference is left for the process
itself and for the kernel socket buffers the cgroup also charges. An
explicit `maxconn` would read simpler, but `strict-limits` is on by
default and HAProxy refuses to start when it cannot raise nofile to
about twice that value. With 1024 soft and 4096 hard nofile `maxconn
4096` exits with `Cannot raise FD limit to 8251`, while `fd-hard-limit
8192` starts with maxconn 2018. `thread-hard-limit` is the CPU limit
rounded up, at least 1. It caps the detected thread count and never
forces more threads than the node has.
Reproduced with Docker Desktop on arm64 (12 vCPU, kernel 6.12) and the
same image, running the rendered config the way the Deployment mounts
it. A larger node is stood in for by adding `nbthread 96` or `nbthread
128` to the `global` section. Memory is `memory.peak` of the container
cgroup.
```sh
cd packages/apps/tcp-balancer && mkdir -p /tmp/cfg
helm template tb . --show-only templates/configmap.yaml | yq '.data["haproxy.cfg"]' > /tmp/cfg/haproxy.cfg
docker run --rm --memory 128m --memory-swap 128m --ulimit nofile=1048576:1048576 -v /tmp/cfg:/usr/local/etc/haproxy:ro haproxy:3.4.4
```
| run | main | this PR |
| --- | --- | --- |
| idle, 12 threads | maxconn 524258, peak 89.6 MiB | maxconn 2035, 1
thread, peak 18.7 MiB |
| idle, `nbthread 96` | maxconn 524127, worker exited with code 137,
OOMKilled | 1 thread, peak 20.8 MiB |
| idle, `nbthread 128` | worker exited with code 137, OOMKilled | 1
thread, peak 22.9 MiB |
| 2035 clients echoing 16 KiB through a backend for 25 s, 12 threads |
worker exited with code 137 about a second after the clients started |
running, peak 83.7 MiB |
One case this PR does not cover is clients and servers that both stop
reading. They fill kernel socket buffers that the cgroup charges, and
with this PR the 128Mi container was still OOMKilled after 1225 such
connections, which maxconn does not bound.
`helm unittest .` passes 13 of 13 with helm-unittest 1.1.2 as CI pins
it. Against the template on main the four new cases fail and the nine
existing ones pass, and each of six template mutations fails at least
one case. `haproxy -c` accepts the render for `t1.nano`, `t1.4xlarge`,
`m1.4xlarge`, a zero CPU limit, the whitelist, `kubernetesAndTalos` and
`tcp-with-proxy` inputs. A full `helm template` differs from main only
in the `global` section and the `checksum/config` annotation, so the
pods roll once on upgrade. I have not run the change on a live cluster.
The template is the same on release-1.6, and that backport is a separate
PR.
### Screenshots
Not a UI change.
### Downstream repositories
One chart template and its unittest suite in
`packages/apps/tcp-balancer`. No values key, schema, default, README or
package change, so no trigger map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(tcp-balancer): HAProxy now takes its fd and thread caps from the replica's memory and CPU limits instead of the node, so a t1.nano balancer is no longer OOMKilled on nodes with many CPUs. maxconn follows the memory limit at about 16 connections per MiB, 2035 on t1.nano, and the pods roll once on upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- TCP Balancer now automatically configures HAProxy file-descriptor and
thread limits based on the selected or explicitly defined resource
settings.
- CPU-based thread limits are rounded up and always maintain a minimum
of one thread.
- Memory-based file-descriptor limits scale automatically with the
configured memory capacity.
- **Tests**
- Added coverage for default, preset-based, explicit resource, rounding,
and minimum-limit scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
432d02e20c |
fix(etcd): stop counting canceled watch streams as failed gRPC requests (#4985)
## What this PR does
The tenant etcd chart has its own copy of
`etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every
`grpc_code` other than `OK` as a failure. `grpc_server_handled_total`
counts a Watch stream once, when it ends, and the streams of a
kube-apiserver that goes away end with `Canceled`. So whenever an API
server on a tenant etcd stops, the Watch method of every member reads as
100 percent failed and the critical rule fires on all of them.
Both rules now count only
`Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`,
the list the etcd mixin has used since etcd-io/etcd#13127 and the one
the platform copy in
`packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The
grouping `BY (job, instance, grpc_service, grpc_method)`, the
thresholds, `for` and the annotations stay as they are, so the alerts
keep their labels.
Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was
created and deleted a few minutes later. vmalert had the critical alert
firing for `grpc_method="Watch"` on all three members with the value
100. At that point
`rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for
Watch with `Canceled`, and the only other non-OK codes were
`FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is
one call each. The same query an hour earlier returned nothing. The
platform copy runs in the same vmalert over the same series, and none of
the firing alerts came from it. Each alert also showed up seven times
there, once per tenant copy of these rules, because the root tenant's
vmalert loads the rules of every tenant namespace. This PR does not
change that.
`hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with
`helm template` and replays it through `vmalert-tool` v1.136.0 pinned by
digest, the same image #4963 uses.
`hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with
seven checks. Watch streams that end only with `Canceled` must not fire
at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25
percent must fire the critical rule at 7 minutes and both rules at 12,
with the labels and the summary pinned. Both together must fire only for
`Range`. Against the rules on main the suite fails 4 of the 7 checks,
all of them on Watch alerts with the value 100. Dropping `Unavailable`
from the list, or grouping by `job` and `instance` only, fails 3 checks
each.
```console
$ bats hack/etcd-grpc-alerts.bats
1..1
ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases
```
The same run on the rules from main prints `not ok 1` and the four
failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes
together with `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`,
`alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like
the migration suites it needs docker, and only the image pull touches
the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run
the new rules in vmalert on a live cluster.
The rules file is the same on release-1.6 and release-1.7. The commit
cherry-picks onto release-1.6 without conflicts, and that backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.
### Screenshots
Not a UI change.
### Downstream repositories
A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and
its test data. No values, schema, README, image or
`ApplicationDefinition` change, and the website does not mention these
rules, so no trigger map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Refined etcd gRPC failure alerts to target specific failure status
codes, reducing alerts for non-failure responses while preserving
existing thresholds and timing.
- Added coverage for canceled watch streams, unavailable range requests,
and combined failure scenarios, including warning and critical alert
evaluations.
- **Tests**
- Added integration validation for rendering etcd alert rules and
running them through the alert-testing tool, including failure handling
when prerequisites are unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
614847976f |
test(harbor): pin the resources of the nginx proxy and the portal (#5025)
## What this PR does #4095 gave the Harbor nginx proxy the `t1.small` preset and moved the portal from `t1.nano` to `t1.micro`, so neither falls back to the LimitRange default of a tenant namespace, 250m CPU and 128Mi. It came without a test. This adds a helm-unittest suite that pins both presets in the nested HelmRelease and checks that they stay the same when the other components are resized, so dropping them or tying them to another component's values fails the suite. Both containers run nginx with `worker_processes auto`, which starts one worker per CPU of the node whatever the container's CPU limit is. With the proxy config the chart renders and `goharbor/nginx-photon:v2.15.1`, 96 workers idle at about 205 MiB, so on a node with many CPUs the proxy is OOMKilled at 128Mi before it serves a request. That is what happened on a v1.6.5 cluster, where both nginx pods of a default Harbor restarted 5 and 6 times within minutes, because `release-1.6` never got #4095. The backport, with this suite and the measurements, is a separate PR. `helm unittest .` in `packages/apps/harbor` passes 47 of 47 with helm-unittest 1.1.2. With the #4095 hunk reverted the three new cases fail and the 44 existing ones pass. Each of seven template mutations fails at least one new case. They drop the nginx block, give nginx `t1.micro` or the portal `t1.nano`, tie nginx to the core values or the portal to the jobservice values, misspell the `resources` key under nginx, and make core ignore its preset, which the resize case checks so that it cannot pass on overrides that never reach the release. ### Screenshots Not a UI change. ### Downstream repositories One new helm-unittest suite in `packages/apps/harbor/tests`. No template, values key, schema, default, README or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Tests** * Added Helm coverage verifying default resource settings for Harbor’s nginx and portal components. * Added coverage confirming nginx and portal memory allocations remain consistent when related service resources are customized. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a26c441d78 |
fix(vm-disk): let CDI adopt the PVC a Velero restore recreates (#4955)
## What this PR does After a Velero restore of a VMDisk or of a VMInstance (`RestoreJob` with the Velero strategy), the disk's HelmRelease fails on its next upgrade and stays failed: ``` Helm upgrade failed for release <ns>/vm-disk-<name> with chart vm-disk@...: failed to create resource: admission webhook "datavolume-validate.cdi.kubevirt.io" denied the request: Destination PVC <ns>/vm-disk-<name> already exists ``` The VMInstance release that references the disk then fails in turn (`Specified disk not exists in cluster`). Seen on a v1.6.5 cluster with CDI v1.66.1, where each failed upgrade is retried and the release history of one such disk had grown past 46,000 revisions. ### Cause The restore controller deletes the disk's DataVolume (`prepareForRestore`) and Velero restores the PVC under the same name, with no owner. The restored release matches its chart and values, so Flux does not upgrade it and the DataVolume is not recreated right away; the next Helm upgrade (a platform update, an edit of the VMDisk) does recreate it, and CDI's webhook refuses it. In CDI v1.66.1 the webhook admits a DataVolume over an existing claim only when `ClaimMayExistBeforeDataVolume` is true (`pkg/controller/common/util.go`): either the PVC carries `cdi.kubevirt.io/storage.populatedFor: <dv name>`, or `AllowClaimAdoption` returns true, which reads `cdi.kubevirt.io/allowClaimAdoption` **from the DataVolume** and otherwise falls back to the `DataVolumeClaimAdoption` feature gate. The Velero strategy's resource modifiers (`createResourceModifiersConfigMap`) put `allowClaimAdoption=true` on the restored **PVC**, where CDI never reads it, and the `CDI` CR in `packages/system/kubevirt-cdi` enables neither the gate nor anything equivalent. Migration 29 creates its PVCs with the same PVC-side annotation. ### Fix The vm-disk chart sets `cdi.kubevirt.io/allowClaimAdoption: "true"` on the DataVolume it renders, on the creation path and on the existing-DataVolume path alike, so the annotation does not flap between upgrades. - A fresh disk is unaffected: no PVC exists when the DataVolume is admitted, and the PVC CDI then creates carries `cdi.kubevirt.io/createdForDataVolume: <dv uid>`, for which `AllowClaimAdoption` returns false, so CDI imports, clones or uploads as before. - An existing DataVolume already controls its PVC, so the annotation changes nothing there; a Helm upgrade only adds it to the DataVolume's metadata, which CDI allows (only the spec is immutable). - After a restore, the webhook admits the recreated DataVolume and the DataVolume controller adopts the claim: `pvcRequiresWork` is false, it takes controller ownership of the PVC and the DataVolume goes `Succeeded` without populating it again, so the restored data is kept. Checked against a v1.6.5 cluster carrying the broken state, by rendering the chart for an affected disk and running `kubectl create --dry-run=server`: the DataVolume without the annotation is refused with the error above, the same one with the annotation is admitted. On that cluster two affected disks were later recovered through the claim-adoption path (the same `AllowClaimAdoption` check): the PVC kept its uid, became controlled by the new DataVolume, and the DataVolume reached `Succeeded` with no import. ### Alternatives considered - **Enable the `DataVolumeClaimAdoption` feature gate in the `CDI` CR.** It works too, but for every DataVolume in the cluster (tenant Kubernetes worker disks, golden images, KubeVirt `dataVolumeTemplates`), each of which would then silently take over any unowned PVC of its name. The per-DataVolume annotation keeps the behaviour to the one chart whose DataVolumes are expected to meet a restored claim, and CDI has said it means to retire the gate (kubevirt/containerized-data-importer#3480) while the annotation is checked first and independently of it. - **Stamp `cdi.kubevirt.io/storage.populatedFor` on the restored PVC** from the resource modifiers. That would also pass the webhook, but only for disks restored by the Velero strategy, not for claims created by migration 29 or by hand, and it would need a controller image change where a chart change suffices. The PVC-side `allowClaimAdoption` rule in `createResourceModifiersConfigMap` (and the one in migration 29) is inert under CDI v1.66.1. This PR leaves the rule and its unit-test assertions in place, and corrects the comments that credited it with the adoption (the collector and `createResourceModifiersConfigMap` in `velerostrategy_controller.go`, and step 5 of `examples/backups/vmi/scenario-user-restore.md`): they now say CDI adopts the restored PVC through the annotation the vm-disk chart sets on the DataVolume. The backports carry only the template and the unittest suite, so these comment and doc edits do not reach them. One behaviour change to be aware of: a PVC named `vm-disk-<name>` left behind without an owner (a restored disk deleted before this fix, whose PVC Velero restored without an owner) is now adopted by a new VMDisk of the same name, with its old contents, instead of blocking that VMDisk forever. That PVC lives in the same tenant namespace. ### Tests - `packages/apps/vm-disk/tests/dv_claim_adoption_test.yaml`: the post-restore state (a restored PVC, no DataVolume) renders the DataVolume through the creation path with the annotation, and the existing-DataVolume path keeps it. Both fail with the template change reverted. `helm unittest packages/apps/vm-disk`: 116 passed. - `examples/backups/vmdisk/05-restore-in-place.sh` (run by the `vmdisk-2-backup-roundtrip` Chainsaw suite): after the restore, the HelmRelease was already reported Ready because no upgrade ran, which is why CI did not catch this. The script now forces a Helm upgrade (`reconcile.fluxcd.io/forceAt` + `requestedAt`), then requires the release Ready, the DataVolume `Succeeded`, and the PVC to keep its restored uid and be controlled by the DataVolume. ### Backports Opened by hand as #4956 (`release-1.6`) and #4957 (`release-1.7`) rather than through the `kind/backport` labels: `examples/backups/vmdisk/` does not exist on either branch, so the bot's cherry-pick would conflict. They carry the template change and the helm unittest suite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: a vm-disk template change (no `values.yaml`, schema, `README.md` or `Chart.yaml` change), a new helm unittest file, and a backup walkthrough script. No app, field, default, CRD, platform value or tooling changes, and no documented workaround on the website is made obsolete (the only mention of `allowClaimAdoption` there is in the archived v0 docs, about a different manual procedure). - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(vm-disk): a VMDisk restored by Velero, alone or as part of a VMInstance, no longer fails its next upgrade with "Destination PVC ... already exists": the DataVolume now carries cdi.kubevirt.io/allowClaimAdoption, so CDI adopts the restored PVC and keeps its data. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved in-place VM disk restores so restored storage is correctly adopted when the disk resource is recreated. * The restore process now waits for the disk to become ready and reports an error if it fails or the restored storage is not retained. * Clarified the adoption behavior in restore documentation and ensured recreated disks use the required adoption setting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
215e582f03 |
fix(hack): keep the backup stack in the vminstance selective install (#5027)
## What this PR does `vminstance` chainsaw suite has two velero round-trips (`vmdisk-2-backup-roundtrip` and `vminstance-2-backup-roundtrip`), both wait for `cozy-default` BSL that `backupstrategy-controller` package renders together with platform Bucket. `hack/select-install.sh` keeps this package only for database and broker suites ([#4022](https://github.com/cozystack/cozystack/pull/4022)), so PR with selection of exactly `vminstance` gets platform without it: no Bucket, no BucketClaim, no BSL, and both round-trips fail on BSL wait after about 190 seconds, before any step of example scripts runs. Seen on the [E2E run of #4955](https://github.com/cozystack/cozystack/actions/runs/39225333408), selected apps there are `vminstance` only and the backup packages are in disabled list. Any edit of `examples/backups/vmi` or `examples/backups/vmdisk` selects `vminstance` and nothing else, so PRs that change those scripts can't run them. Fix is one more entry in `suite_runtime_sources`, the table that already keeps the stack for database suites. Install closure of `vminstance` grows by `backup-controller`, `backupstrategy-controller`, `bucket-application`, `velero`, `metrics-server` and `monitoring-agents`. Bats test for backup suites now includes `vminstance` and fails without the script change. `hack/select-install_test.bats` and `hack/select-e2e_test.bats` pass (110 tests), `hack/select-install.sh --validate` is clean. Own E2E run of this PR doesn't cover the fix, because `hack/*.sh` escalates to full suite and database suites install the backup stack anyway. Real check is re-run of E2E for [#4955](https://github.com/cozystack/cozystack/pull/4955) after merge, it selects `vminstance` alone. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff, it is one selector table entry and its bats test, no file under `hack/` moved or renamed and no make target changed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(hack): E2E install for `vminstance`-only selection keeps the backup stack, so VMInstance and VMDisk backup round-trips run instead of timing out on BSL wait. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **Bug Fixes** - Virtual machine instances now correctly include the backup strategy component required for backup-related functionality. - Installation checks now consistently validate this requirement, including when backup features are disabled. This improves setup reliability and helps prevent incomplete or misconfigured virtual machine deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d78897f842 |
fix(cluster-api): resync capi-operator every minute (#4939)
## What this PR does After an upgrade restarts capi-operator, a Cluster API provider can keep `Ready=False` for about 20 minutes while its controller is already running, and `capi-providers-core` times out waiting for the CoreProvider. The bundled operator v0.19.0 drops create events of provider Deployments it finds running after a restart (kubernetes-sigs/cluster-api-operator#1031), so it looks at them again only when one changes or on informer resync, and the first resync after a restart comes two sync periods later, about 20 minutes with the default 10m. This sets the operator `--sync-period` to 1m. On a test cluster a provider left `Ready=False` across an operator restart recovered 2m02s after the operator took its lease, the operator with the default period had not recovered after more than 4 minutes. A resync is served from the operator cache and writes nothing while providers are in sync: no provider object was updated over five resyncs, about 15 log lines a minute, 2m CPU and 26Mi memory. The vendored chart has no value for this flag, so a small package patch (reapplied by `make update`) adds one, and a test holds the rendered argument. The upstream fix is in cluster-api-operator v0.27.0, bumping to it is a bigger change and is left for a later release. Mitigates #4926. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(cluster-api): resync capi-operator every minute so a provider left not ready across an operator restart recovers in about two minutes instead of about twenty ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * The Cluster API operator now refreshes its cached state every minute. After an operator restart, provider readiness status can recover sooner instead of remaining stale for a longer period. * Providers that are already in sync are not written back to the API during these cache refreshes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Mattia Eleuteri
|
b5d7e43c49 |
docs(backups): say CDI adopts a restored PVC through the DataVolume
The restore controller and the user-restore example described the allowClaimAdoption annotation its resource modifiers put on restored PVCs as what lets CDI adopt them. CDI v1.66 reads that annotation only on the DataVolume; on a PVC it reads nothing but createdForDataVolume. Adoption rests on the vm-disk chart annotating the DataVolume Flux recreates, and a reader trusting the PVC rule could rely on it for a restore without such a DataVolume, or drop the DataVolume annotation as redundant. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
Aleksei Sviridkin
|
d24b92fe1d |
docs(agents): define where a new document belongs (#3777)
## What this PR does `docs/` had collected component guides that belong on the website, standalone oidc, gpu and vm networking documents written inside the feature PRs that added the code and never routed anywhere a user looks. Nothing said where documentation should go, and in particular nothing mentioned that the website appends a whole package README onto a published page, so a contributor with package scoped prose had no reason to use that route. This replaces the four line "Documentation is organized as follows" list in `docs/agents/overview.md` with a destination table and the test that separates `docs/` from the website: whether a non-documentation file has to link to the document by path. It also says that a merged document records current behaviour and its limitations, not the development process around it, which is the "What is NOT in Phase N" and "this PR" residue that motivated the whole thing. One such residue dropped while here, `docs/operations/backup-classes.md` described a pin as one "this PR removes". Moving the documents themselves is a separate PR. ### Downstream repositories - [x] No downstream repository is affected by this change Walked the trigger map against the diff: this touches `docs/agents/overview.md` and `docs/operations/backup-classes.md` only. No package added, renamed or removed, no `values.yaml` or `values.schema.json` field, no `packages/core/platform/values.yaml` key, no bundle or variant, no release asset name, no `ApplicationDefinition` semantics, no Talos pin, and no change to `cozyvalues-gen`, `cozypkg` or a package Makefile. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Added guidance for choosing where new documentation belongs, including package READMEs, the website, repository docs, API design documents, and community resources. - Clarified package README publishing and sync conditions, when repository documentation is warranted, and where to document feature limitations and development-process notes. - Clarified that documents referencing unmerged pull requests or unreleased dependencies need a tracking issue. - Updated the CloudNativePG upgrade note to explain that the CRD version skew came from an image tag pin outrunning the vendored chart CRDs; the pin has since been removed. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yan Bondarenko
|
1d8bea039b |
test(harbor): check that the resized components reach the release
The case that resizes the other components looked only at nginx and the portal, so it would keep passing if those overrides stopped reaching the nested release, and then it could not catch nginx or the portal following another component's values. It now checks the core, registry and jobservice limits it sets and leaves trivy alone, as nothing in the case reads it. The portal case also checks its memory request, as the nginx case does. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Myasnikov Daniil
|
9a782f697a |
fix(hack): keep the backup stack in the vminstance selective install
The vminstance Chainsaw suite carries the VMInstance and VMDisk Velero round-trips. Both build the cozy-default BackupStorageLocation and the platform bucket from the backupstrategy-controller package, but the selective install retained that package only for the database and broker suites. A selection of vminstance alone therefore installed a platform with no Bucket, no BucketClaim and no BSL, and both round-trips timed out on the BSL wait before running a step of the example scripts. Every pull request whose selection is exactly vminstance hits this, and that includes each edit to examples/backups/vmi and examples/backups/vmdisk, which select that suite on purpose. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Yan Bondarenko
|
7c0634586b |
test(harbor): pin the resources of the nginx proxy and the portal
Both containers run nginx with worker_processes auto, so their idle memory follows the CPU count of the node, whatever their own CPU limit is. In a tenant namespace a container without resources gets the LimitRange default of 128Mi, which the proxy outgrows on a node with many CPUs before it serves a request. The presets that keep both out of that default have no test, so dropping them or tying them to another component's values would go unnoticed. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
myasnikovdaniil
|
3cd261277e |
fix(backups): keep the backupstrategy-controller Helm release under the Secret size limit (#4941)
## What this PR does Fixes a size regression from #4929. Helm stores each release revision in one Secret and the apiserver refuses a Secret over 1 MiB. With all default Strategy CRs rendered the `backupstrategy-controller` release crosses that limit, so the forced upgrade that creates them fails with `Secret "sh.helm.release.v1.backupstrategy-controller.vN" is invalid: data: Too long: may not be more than 1048576 bytes`, the CRDs stay and the default strategies never appear. The release was already within about 1 KB of the limit before #4929, which added tests to the chart. The strategy CRDs are 2.2 MB of YAML and Helm stores them twice per revision, as chart files and in the rendered manifest. Almost all of it is the upstream `PodTemplateSpec` schema inlined by the Altinity, Job and FoundationDB strategies. Kafka, RabbitMQ and Redis already mark their `Template corev1.PodTemplateSpec` as `Schemaless` and `PreserveUnknownFields`, this PR does the same for `AltinitySpec.Template`, `JobSpec.Template` and `FoundationDBTemplate.BackupDeploymentPodTemplateSpec`. Go types are unchanged and every description is kept. The CRDs go from 2.2 MB to 317 KB and the largest release revision from over the limit to about 250 KB, 24% of it. The size rationale now lives once in the package doc, the field docs of the six schemaless templates share one short line. What changes for users: the apiserver no longer validates these three fields and `kubectl explain` no longer lists children under them (`kubectl explain pod.spec` still documents them), same as for Kafka, RabbitMQ and Redis today. Unknown keys are stored instead of pruned, the typed decode in the drivers drops them either way. Labels and annotations under `template.metadata` were rejected by the strict v1.6 schema and are now accepted and kept. A Go template in a non-string slot never worked, it was rejected on apply, now the apiserver accepts it and the typed decode in the controller is what rejects it, as for the other three kinds (I did not run that case live). The drivers read these fields through the non-strict client and `internal/template` rewrites only string leaves, nothing under `internal/backupcontroller` relies on pruning. Go tests for `internal/backupcontroller`, `internal/template` and `api` pass. `hack/strategy-crd-schemaless.bats` fails on any strategy CRD that inlines a pod schema and prints the property path and the two markers to add. `hack/helm-release-size.bats` estimates the stored release of every chart under `packages/` the way Helm encodes it (release JSON, gzip, base64) and fails above 90% of the limit, plus the worst case of this chart. The estimate runs 1 to 2 percent above real Secrets. A chart has to render offline to be measured with its manifest: the 36 charts that do not are listed in `NO_OFFLINE_RENDER` and measured on files and templates only, any other render failure fails the test, and so does a listed chart that starts rendering. Checked on a throwaway kind cluster: v1.6.4 upgrades to this branch and the forced follow-up creates all default strategies, a fresh install works, strategy objects created under the old schema stay valid, a Job strategy with templated args and pod template labels is accepted. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): the backupstrategy-controller release no longer fails with "Secret ... Too long" on the upgrade that creates the default backup strategies, the Altinity, Job and FoundationDB strategies store their pod templates schemaless like Kafka, RabbitMQ and Redis ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Backup strategy templates for Altinity, FoundationDB, Jobs, Kafka, RabbitMQ, and Redis now preserve fields not defined in the API schema. The API server no longer validates these templates against the pod schema; validation occurs when the driver creates the workload. This lets strategies include pod-template fields without requiring those fields to be listed in the strategy schema. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
c6fcb4e7c6 |
fix(ci): dispatch frozen-line builds with GITHUB_TOKEN (#4888)
## What this PR does The first RC freeze hit HTTP 403 when dispatching `build-release.yaml` because the CI app has no Actions write permission ([run](https://github.com/cozystack/cozystack/actions/runs/38229949424)). I switched this dispatch to `GITHUB_TOKEN` with job-scoped `actions: write`. Tag and branch pushes still use the app token. Verified with `make unit-tests`, zizmor 1.26.1 and the commit trailer check. The regression test fails on the old workflow and passes with the fix, including warning and manual recovery on dispatch failure. Live dispatch with `GITHUB_TOKEN` has not been tested. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): use GITHUB_TOKEN to start the initial packages build when freezing a release line ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated release automation to use appropriate permissions and authentication for CI triggers. * Improved warnings and recovery guidance when automated package builds cannot be started. * **Tests** * Added automated checks covering release dispatch permissions, branch targeting, successful execution, and non-blocking failure handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
2e4783600b |
docs(release): point changelog check B at the Talos installer profile (#4959)
## What this PR does Check B loops over three values files and the first one, `packages/core/talos/values.yaml`, is an empty file, so nothing is ever printed for Talos. Talos version is `version` in `packages/core/talos/images/talos/profiles/installer.yaml` (package Makefile reads it as `TALOS_VERSION`), this PR points the loop there. I ran old and new loop on real tags. On `v1.5.4..v1.6.0` the Talos bump from `v1.13.0` to `v1.13.6` was not shown before and is shown now, on `v1.6.1..v1.6.2` and `v1.6.4..v1.6.5` output is the same. Found it while reviewing #4952, which fixes the Dockerfile pathspec right below the loop. The two PRs merge cleanly in any order. I left `head -40` alone, it also hides the new `talos.version` block of `packages/apps/kubernetes/values.yaml` on `v1.5.4..v1.6.0`, but that is a separate decision. ### Screenshots N/A: documentation-only change, no UI impact. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note docs(release): point changelog check B at the Talos installer profile, a Talos version bump was not shown by it before ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated changelog verification to check the installer configuration; checks for installer and Kubernetes values remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a6cb84673e |
docs(release): check changelog claims against the code (#4936)
## What this PR does Changelog generator sees only `docs/agents/changelog.md`, two inline prompts and `AGENTS.md`. Failure modes and pre-publish checks live in `docs/release.md`, which it never reads. The prompt has detailed rules for authors and PR numbers and none for checking a claim against the code. I reviewed the generated v1.7.0 draft against the code and found 22 statements wrong or missing, while all authors and PR references were right. Several were in breaking changes: a talm fix credited to a talm release that doesn't contain it, worker pools said to be adopted without churn while Talos workers are rolled once because of the hash-named TalosConfigTemplate (#3523), migrations said to run on a variant that has them disabled. Three changes: - `docs/agents/changelog.md`: new section 8b about verifying claims against the code. Identifiers, migration behaviour and variant scope come from the tree at the upper-bound tag, required actions come from release notes and newly added upgrade sections, not only from `!` commits. Section 9 now asks to run the `docs/release.md` checks and `hack/validate-changelog.sh` on the written file before exit. - `.github/workflows/changelog-generate.yaml`: refresh `docs/release.md` and `hack/validate-changelog.sh` from the dispatch ref, next to the existing `changelog.md` refresh. Job checks out the rc tag detached, so without this generator runs older copies from that tag. Permissions, tokens and tool flags are unchanged. - `hack/validate-changelog.sh`: reject an entry cut off with an ellipsis right before its attribution, and any link to `cozystack.io/docs/next/` (live site answers 404). Reason names the offending lines in one line, because workflows quote it into a single annotation. How I checked it: - Old validator accepts the original v1.7.0 draft, new one rejects it on three truncated entries (lines 335, 393, 518) and, after those are repaired, on the `/docs/next/` link (line 25). All 29 shipped changelogs from v1.3.0 to v1.6.4 still pass. - New contract test plants each defect into a shipped changelog, checks the reason names the line, and checks that near misses real changelogs carry still pass (`content/en/docs/next/` in prose, ellipsis inside a sentence, three dots of a compare link). It goes red on the old validator and on widened patterns. - Overlay test pins both new copies and the cone-mode sparse checkout that brings `docs/release.md` in. It goes red if either copy is removed, cone mode is turned off or `docs/agents` is dropped from sparse paths. - I ran every command the new prompt rules prescribe on v1.6.0..v1.7.0-rc.1, each one finds an error the draft shipped. - `make bats-unit-tests bats-posix-compat-tests` pass (2424 and 1120), zizmor and actionlint are clean on the workflow. Not included: a script that gives the generator component versions and migration headers, and a networked check for references, authors, link liveness and companion-release containment. Both are separate changes. Things I noticed and left alone: - validator summary in the Phase 2 section of `docs/release.md` doesn't list the two new rejections - `docs/agents/changelog.md` runs `docs/changelogs/hack/check-optional-repos.sh`, but the script lives at `hack/check-optional-repos.sh` and still resolves repo root from the old location - a changelog the validator rejects is still not uploaded as an artifact, so rejection costs a regeneration or a hand-commit, that is why section 9 has the generator run the validator first - current v1.7.0 draft still has the three truncated entries, a promotion that picks it up will discard it and generate a new one unless the entries are rewritten - three unrelated bats tests (in `check-gpu-recording-rules.bats` and `promote-rewrite-tags_test.bats`) fail under en_US.UTF-8 collation on main too, they pass under C.UTF-8 that runners use, I ran the suite that way ### Screenshots Not applicable, there is no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I walked the trigger map in `docs/agents/contributing.md` against each changed file: - `docs/agents/changelog.md` is the changelog prompt, not the commit convention or the PR template that ccp and community restate. Its list of release-window repositories is unchanged. - `.github/workflows/changelog-generate.yaml` is not `tags.yaml`, the release-prep workflow ccp relies on. - `hack/validate-changelog.sh` is changed in place, not moved or renamed, and it is not `hack/package.mk` or `hack/update-crd.sh`. - `hack/release-changelog-contract.bats` is a test. No package, values key, CRD, release asset, make target or node prerequisite changes. Code search across the organization finds no reference to any of these files outside this repository. ### Release note ```release-note docs(release): check changelog claims against the code. The changelog generator now verifies operator-facing statements against the code and runs the release checks on its own output, and the changelog validator rejects entries cut off with an ellipsis and links to cozystack.io/docs/next/. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Release guidance now emphasizes verifying operator-facing claims against the release version and carrying required actions into upgrade instructions. * Documentation links in changelogs should point to versioned documentation. * **Chores** * Changelog validation now flags truncated entries and links to unversioned “next” documentation. * Release guidance and validation checks are refreshed automatically during changelog generation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
f69958b15d |
docs: move the operator guides out of docs/ to where readers find them (#3780)
## What this PR does
Five documents under `docs/` were operator facing guides written
alongside the feature PRs that added the code, and left in the source
tree where no user looks. They are on the website now, so this deletes
them here and repoints every reference.
```
docs/oidc-tenant.md -> kubernetes/oidc-authentication
docs/oidc-grafana.md -> operations/services/monitoring/oidc-authentication
docs/gpu-vgpu.md -> virtualization/vgpu
+ operations/troubleshooting/gpu-operator-host-driver
docs/vm-external-vlan.md -> networking/vm-external-vlan
docs/hubble-observability.md -> networking/hubble
```
`docs/vm-external-vlan.md` needed splitting, not moving. Its "Staging
the reference CNI plugins" section is a contract,
`hack/cni-plugins-staging-contract.bats` greps six literal sentences out
of it and asserts them against the multus Dockerfile, the daemonset
manifest and the platform values. That half is package documentation so
it moves verbatim into a new `packages/system/multus/README.md` and the
suite reads it there, the operator guide goes to the website. The bats
messages name the README now, so a future failure points at the file
that has to change.
References repointed in the `@param` blocks, template comments and test
comments that named the guides. The generated files come from `make
generate` with cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`. Where a reference points at text
that exists only on `next/` of the website (the aggregated API servers
section of the tenant OIDC guide), the comment names the page and the
section instead of a `v1.6` URL, which would not contain it.
Two changelog links in `docs/changelogs/v1.6.0.md` repointed as well.
The wording is untouched, only the location of the documents it names
has changed.
One bit of process framing dropped on the way past, per #3777:
`bundles/system.yaml` said GrafanaAdmin promotion was "out of scope for
Phase 1" when it is simply not implemented.
Tests: 10 cni-plugins contract, 47 install-wait, 346 kubernetes chart,
22 multus chart, 301 monitoring chart, rd-presets-check.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#648 (the four guides),
cozystack/website#754 (edits made to the originals in `docs/` after the
import), cozystack/website#647 (grafana guide made reachable in v1.6),
cozystack/website#596 (tenant kubectl guide, rebased)
Walked the rest of the trigger map against the diff. No provider
follow-up: the kubernetes `values.schema.json` diff is two `description`
strings and the monitoring one is one, with no field added, removed or
renamed, no enum, no default, and no `kind`, `plural` or
`release.prefix` change. `packages/core/platform/values.yaml` changed in
a comment only and no key, so the hand written platform-package table on
the site is unaffected.
Please merge after cozystack/website#754 and cozystack/website#647. The
`@param` URLs only resolve once those pages deploy, and
cozystack/website#754 carries the edits made to the guides since the
import, which deleting them here would lose.
### Release note
```release-note
docs: operator guides for tenant Kubernetes OIDC, Grafana OIDC, NVIDIA vGPU, VM external VLAN and Hubble now live on cozystack.io instead of the docs/ directory of this repository, and in-repo references point there. The CNI plugin staging contract that shared the VLAN guide moved to packages/system/multus/README.md.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated configuration and test references to link to the published
Cozystack documentation.
* Added Multus guidance covering staged CNI plugins, configuration,
troubleshooting, and rollout behavior.
* Removed outdated local guides for GPU/vGPU, Hubble observability, OIDC
authentication, and external VM VLANs.
* Clarified that server-level Grafana administrator promotion is not
currently supported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Yan Bondarenko
|
20545d00a5 |
fix(tcp-balancer): cap HAProxy fds and threads by the pod's own limits
Without a global section HAProxy 3.4 sizes itself from the node. maxconn comes from the nofile limit, cut to the built-in fd-hard-limit of 1048576, so about 524k, and the thread count is the number of CPUs the process may run on, whatever the CPU limit says. With one thread that alone idles at about 85 MiB, and every thread adds about 0.6 MiB, so a t1.nano replica with its 128Mi limit runs on a small node and is OOMKilled at startup on a node with many CPUs. fd-hard-limit is now 32 per MiB of the memory limit, from which HAProxy derives about 16 connections per MiB, 2035 for the t1.nano defaults. That leaves 64 KiB per connection against the roughly 35 KiB HAProxy budgets per connection itself when it is given a memory limit with -m, and the rest covers the process and the kernel socket buffers charged to the same cgroup. It caps the derived value instead of setting maxconn, because an explicit maxconn above half the hard nofile limit stops HAProxy at startup (strict-limits is on by default), while fd-hard-limit still follows a lower system limit. thread-hard-limit is the CPU limit rounded up, at least 1. It only caps the detected thread count, so a node with fewer CPUs is not forced to run more threads than it has. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
myasnikovdaniil
|
1f92bcd20e |
fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings (#4914)
## What this PR does Follow-up to the flux-plunger uninstall fence: the points the last review round raised and left for after the merge, plus four found by a later independent review. None changes the fence's design; each closes a gap in what it reports, how long it waits, or what its documentation says. **Pre-delete hook.** - A release being deleted that only the fence still holds is no longer unfenced in the same step that lifts its suspension. helm-controller runs the uninstall of a release being deleted once it is no longer suspended, but removing the fence at once deleted the object before helm-controller could reconcile it. The hook now keeps that fence while helm-controller handles the generation the lifted suspension wrote, until shortly before its retry window closes, and names a release it still had to let go of as before. A release whose state cannot be read keeps its fence for the wait the same way, rather than being let go of on one failed read. - A release being deleted while still suspended, by anyone but flux-plunger, was reported as cleanly released, although helm-controller skips the uninstall of a suspended release. It is now named as deleted without an uninstall and counted with the others. - Each release gets its own share of the retry window, capped at the window's end. One release that could never be read used to take the whole window and leave every release listed after it a single attempt. The share is a cap, not a floor of attempts: a floor would let enough failing releases run the Job past its deadline, which kills it before its report is written. - The hook stops, waits for and lists again only for the flux-plunger of its own release. Another one (a second installation under another release name, changed labels, a binary outside the cluster) is not stopped by the hook, keeps the fences it sets and gives them up itself; the README now says so instead of the hook re-listing unconditionally, which would not close that case and would fail the uninstall for as long as a second installation is recovering anything. - helm-controller's finalizer, which the hook reads to tell whether a release being deleted is still held, is tied to the constant the controller uses, as the fence already was. Two comments are corrected: the retry window counts from the script's start, not the Job's, and kubectl scale's internal conflict retry never runs here. **Shutdown drain.** - A drain deadline of zero or less is refused at startup, and the chart's lowest accepted grace period is held to the same constants as the drain subtraction, so raising the manager's stop bound cannot make the chart render a deadline that is already spent. The timings the README and the values comment state are held to those constants too. - Fences left behind are reported as an error naming each release, not an Info line; a drain that never got a listing answered says so instead of reporting a count of -1; a flux-plunger that never led no longer claims it released every fence. - The drain's Events fold repeats into one counted Event per release and reason, as the manager's recorder does, instead of one object per pass. The fold is a patch, the verb the chart grants on events, and every write is bounded by the drain's deadline. It does not add a per-release Event for fences left at the end: by then the drain's deadline is spent, and writing one would reach past it. - Tests now hold the outcome line, the retry of a release whose fence cannot be given up yet, and the pause between passes, none of which any suite noticed when removed. **Controller.** The `SuspensionNotOwned` and `TerminatingWhileSuspended` Warnings were keyed on the processed-version annotation, which stays on every release flux-plunger ever recovered, so a later restore, migration or operator suspension of a healthy release drew a Warning telling the reader to resume it, and a migration's deliberate suspend-then-delete was called an accident. Both now key on the state a leftover actually freezes the release in, `Ready=False` with `has no deployed releases`, since helm-controller rewrites no condition of a suspended release; the README's candidate query follows. Two comments that claimed more than holds are qualified. **Chart and README.** A `shutdownGracePeriodSeconds` of a million or more now gets the range message rather than being refused as a fraction. The README gives the recovery steps in the order the controller runs them (the annotation is written before the Secret is deleted), and names the window in which a backup restore started during a recovery is unsuspended under it, since the etcd, CNPG and Velero restore steps write `spec.suspend` only when it differs. ### Testing Each change was written test first: a test that builds the binary and runs it with a deadline of zero, helm-unittest for the chart, unit tests with a captured logger and the fake client for the drain's outcome, Events and passes (the Event fold under a client that refuses the update verb, as the chart's role does), table tests for both Warnings, and the bats suite for the hook, whose fake kubectl now models helm-controller handling a release some reads after its suspension is lifted and a state read that fails. Every fix was then reverted on its own, and the suite went red each time. The envtest suites pass unchanged. The README's candidate query was run against HelmReleases read from a live cluster, with a stuck, a healthy, an owned and an unsuspended variant, and lists only the stuck one. ### Screenshots Not applicable: no UI change. ### Downstream repositories The trigger map was walked against the diff: `internal/controller/fluxplunger`, `cmd/flux-plunger`, the `packages/system/flux-plunger` chart (templates, tests, README) and its bats test under `hack/`. The website's FluxPlunger section in `guides/concepts.md` draws the recovery with the Secret deleted before the annotation is written, which is the order before the fence change; it needs the same correction as the chart README. That change is held as a draft until v1.7.0 is released and the website's `next/` is re-initialized: the fence change is not part of release-1.7, so the corrected order would be wrong for v1.7 docs, which are copied from `next/`. No other repository is reached by this diff. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/752 (draft, merge after v1.7.0) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(fluxcd): flux-plunger's pre-delete hook now lets helm-controller run the uninstall of a release being deleted before it removes the release's fence, gives each release its own share of its retries, and reports releases deleted under another manager's suspension; the shutdown drain reports the fences it leaves as errors and refuses a deadline that leaves no time ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Shutdown drain timeouts must be positive, and invalid values now stop startup with a clear error. * Uninstall cleanup waits for Helm to handle releases before removing their fences, while bounding retries and reporting releases that remain incomplete. * Recovery and suspension warnings now reflect the release’s actual failure state, and recovery can retry stale storage cleanup against the same revision. * **Documentation** * Clarified shutdown grace-period requirements, uninstall behavior, and guidance for backup restores and existing suspensions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d1d4e92acb |
feat(release): sign the release assets and attach a source SBOM (#4861)
## What this PR does Implements the release-asset part of #4759: a release cut from a tree that has this change gets an SBOM of its source and one keyless signature that covers all of its assets. These are the two CLOMonitor checks still failing for this repository, `sbom` and `signed_releases`, after #4757 and #4899 brought the project score to 96.69. `hack/upload-assets.sh` now collects its asset list into an array and hands it to the new `hack/sign-release-assets.sh` before the first upload. The script writes three files into `_out/assets`, and they are uploaded with the rest: | File | What it is | |---|---| | `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over `git archive HEAD` (the committed release tree, without `_out/`, `.release-tooling` or runner leftovers) and the unpacked `cozypkg-*.tar.gz` binaries. It does **not** describe the container images that `packages/` pins by digest (their SBOMs are #2937, A2 in #3278) or the Talos ISO, disk, kernel and initramfs images. It is named by what it covers, so `cozystack-sbom` stays free for an SBOM of the release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset name. | | `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the SBOM. | | `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign sign-blob` over the checksums, keyless, signed with the job's GitHub OIDC identity. | Before it returns, the script runs the same `cosign verify-blob` that `docs/release.md` gives users, with `--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`: `tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from `main`. A signature that check would reject therefore fails the job before anything is uploaded. It also refuses its own three outputs as inputs: `promote-rc.yaml` downloads every rc asset, and a stable release has to get freshly computed checksums and a fresh signature, since promotion restamps several assets. Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, the two jobs that call `make upload_assets`, get `id-token: write` and two setup steps: - syft v1.54.1 from its release tarball, checked against a committed SHA-256 (`c069905b…`, the release's checksums entry; the checksums file verifies against anchore/syft's keyless release signature). Not `anchore/sbom-action/download-syft`: pinned by commit, it still runs syft's `install.sh` fetched from a tag at run time, in a job that holds the app token, the GHCR login and `id-token: write`. v1.54.1 is past GHSA-mw2c-m758-9v5q (fixed in v1.52.0). `hack/ci-tool-installs-contract.bats` now fails on `anchore/sbom-action`, on a syft install script, on a syft download without `sha256sum -c` and on version or checksum drift between the two workflows. - `sigstore/cosign-installer` v4.1.2, pinned by SHA, with `cosign-release: v3.1.3`: the version `docs/release.md` asks users for, so the job checks its own signature the way they will. The action verifies that binary against cosign's keyless and KMS bundles with its digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats` holds the action commit and the release equal in both workflows and fails on a step without `cosign-release`. `docs/release.md` gets a "Release signatures and SBOM" section with the user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line), the two commands joined with `&&`, what the SBOM covers and what the certificate identity proves. It now also says, at each place `promote-rc.yaml` is run, that it is dispatched from `main` only. ### Decisions for the release owners - **Fail closed.** If Fulcio or Rekor is down, the release job fails and nothing is uploaded. I chose that so that no release goes out unsigned. If you would rather publish unsigned than block, the change is a `|| echo ::warning::` around the call in `upload-assets.sh`. - **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make upload_assets` from the rc checkout, not from `.release-tooling`. The first signed stable release is therefore the promotion of the first rc tagged after this merges. `release-1.6` gets signatures only if this is backported. CLOMonitor reads the latest release, so the score moves only when that release carries the files. - **Certificate identity pins the refs** (decided in review): `tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A `promote-rc.yaml` dispatched from another branch fails the job's own check at the upload step, after the staging branch, the `promotion-*` candidate and the empty draft release exist; nothing in `parse` refuses a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs, which `docs/release.md` says. - **Follow-ups after the first signed release** (#4759 stays open for them): - `project.documentation.signature-verification` in `security-insights.yml` (#4757); - the SBOM row in `docs/security/self-assessment.md`, which says "Not published at this time"; the image caveat above stays in it. I left both alone so that they do not describe releases that don't exist yet. ### How it was checked - Bats 1.14.0 (the version CI installs), with gh, syft and cosign stubbed: - `hack/sign-release-assets.bats`, 8 tests: what the checksums cover; that syft sees the committed tree and the cozypkg binaries but not untracked files; that the verify identity accepts `tags.yaml` on a `v*` tag and `promote-rc.yaml` on `main`, and rejects a fork, another workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and `promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that `docs/release.md` shows the same regexp as the script; that a failed verification fails the script; that it refuses a missing asset, assets split over two directories (with a same-named file in both, so the test fails if the directory check is removed) and its own outputs as input. - `hack/upload-assets.bats`, 4 tests, new: the exact upload list and order (the assets, then the SBOM, the checksums and the bundle); that the checksums name every upload but the last two; zero `gh` calls when verification fails or an asset is missing. The old `upload-assets.sh` fails all four, a version that signs after uploading fails the last two. - `hack/ci-tool-installs-contract.bats`, 8 tests. Putting `download-syft` back, dropping the `sha256sum -c` line, letting `SYFT_VERSION` drift between the workflows, dropping `cosign-release` from one step, letting it drift, unpinning the installer or giving one step another installer commit each fails it. - Putting the old any-ref regexp back in the script fails the identity test and the docs tie; putting it back in the docs alone fails the tie. - The meta-suites `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract` pass: 90 tests across these eight files. - The pinned regexp gives the same accept/reject results under `grep -E` and Go's `regexp` (what cosign uses). cosign-installer's install script at the pinned commit, run as written with `cosign-release: v3.1.3`, verifies the binary's keyless and KMS bundles and installs v3.1.3 (its SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`, `--certificate-identity-regexp` and `--certificate-oidc-issuer` are listed, not deprecated, in cosign v3.1.3's help. - The `Set up syft` step's script, run as written with `RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version` reports 1.54.1. A real syft v1.54.1 run through `hack/sign-release-assets.sh` (cosign stubbed) over this tree and the v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3 document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a purl; 60 come from the cozypkg binary. - cosign behaviour behind the docs, with a key-signed legacy bundle whose `cert` holds the bare public key, against the documented identity regexp (tlog ignored, since an attacker can upload their own entry): v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it. `--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs give the minimum version and not the flag. A real keyless protobuf bundle (syft's) verifies on v2.6.5 and v3.1.3. - zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with `.github/actionlint.yaml`) reports the same 13 pre-existing info-level shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh` passes. - Not checked: a real keyless signature. It needs the OIDC token of a job in this repository, so the first rc after the merge is the end-to-end test. Cutting an `-alpha.N` first would exercise it without touching an rc. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I checked the trigger map file by file. The only entry `hack/upload-assets.sh` can trigger is renaming a release asset, because the website install pages hardcode asset names. This PR adds three assets and renames none of the released ones, `cozypkg-checksums.txt` included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in review, before any release carried it). The workflow and docs changes match no entry. ### Release note ```release-note feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later. ``` Refs #4759 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Releases now include an SPDX software bill of materials, SHA-256 checksums, and a keyless Sigstore signature bundle. * Release assets are verified against the expected workflow identity before upload; uploads stop if verification fails. * **Documentation** * Added guidance on release signatures, SBOM coverage, asset verification, and which branch dispatches release workflows. * **Tests** * Added coverage for generated release files, upload behavior, and signing and verification failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yan Bondarenko
|
1ca7c0d891 |
fix(etcd): stop counting canceled watch streams as failed gRPC requests
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted every code other than OK. A Watch stream is counted once, when it ends, and the streams of a kube-apiserver that stops end with Canceled. So whenever an API server on that etcd goes away, the Watch method of every member reads as 100 percent failed and the critical rule fires on all of them. The rules now count the codes the etcd mixin counts since etcd-io/etcd#13127, the same list the platform copy in monitoring-agents already uses. The grouping by method stays, so the alert labels do not change. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Marian Koreniuk
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
Myasnikov Daniil
|
cc7f534f5e |
docs(release): point changelog check B at the Talos installer profile
packages/core/talos/values.yaml is an empty file, so the first path of the check B loop never produces a diff and a Talos bump goes unnoticed by the release verification. The Talos version is the version field of packages/core/talos/images/talos/profiles/installer.yaml, which the package Makefile reads as TALOS_VERSION. Between v1.5.4 and v1.6.0 it moved from v1.13.0 to v1.13.6 and check B printed nothing for it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Mattia Eleuteri
|
b4f6fb67db |
fix(vm-disk): let CDI adopt the PVC a Velero restore recreates
A Velero restore of a VMDisk or a VMInstance deletes the disk's
DataVolume and restores the PVC under its name, with no owner. The
restored release is in sync, so nothing recreates the DataVolume until
the next Helm upgrade, typically a platform update. That upgrade is
refused by CDI's admission webhook ("Destination PVC ... already
exists") and the release stays failed, followed by the VMInstance
release that looks the disk up.
CDI admits a DataVolume over an existing claim only when the
DataVolume carries cdi.kubevirt.io/allowClaimAdoption or the
DataVolumeClaimAdoption feature gate is on. The restore controller
puts the annotation on the restored PVC, where CDI never reads it, and
the CDI CR does not enable the gate. Setting it on the DataVolume
keeps the scope to vm-disk instead of every DataVolume in the cluster,
and does not depend on a gate CDI intends to remove. A claim CDI
created for this DataVolume carries its uid in createdForDataVolume
and is never treated as adoptable, so a fresh import is unaffected.
The VMDisk backup round-trip now forces an upgrade after the restore,
which is what reaches the failure, and checks that the DataVolume
adopted the restored PVC instead of replacing it.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-by: LLM
|
||
|
Myasnikov Daniil
|
7ad7d39349 |
test(hack): keep every chart's stored Helm release within budget
A chart can grow past the 1 MiB Helm release Secret without any test noticing: it renders, lints and installs, and only the upgrade whose manifest is large enough fails, on a live cluster, with a release that stays failed. backupstrategy-controller sat under 1 KB from the limit before it crossed it. Estimate the stored size the way Helm encodes it (release JSON with the chart's templates and files, gzip, base64) and fail any chart under packages/ above 90 percent of the limit, plus the largest revision of backupstrategy-controller with every default Strategy rendered. The estimate counts files a .helmignore names, because the production path stores them. A size measured without its manifest is a lower bound that passes for the wrong reason, so every chart must render offline except an explicit list of charts that need cluster state, and the worst-case render must succeed and contain the default strategies before it is measured. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
513b5088a8 |
test(backups): fail on a strategy CRD that inlines a pod schema
The release-size budget only reports that the backupstrategy-controller release grew too large, not which field did it, and three strategies once inlined the upstream pod schema before anyone noticed. Check the generated strategy CRDs for an inlined PodSpec and name the property path, with the markers that fix it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
2f6c6a27b8 |
fix(backups): store the Altinity, Job and FoundationDB pod templates schemaless
backupstrategy-controller renders its strategy CRDs from templates, so Helm stores each of them twice in every release revision, and a revision is one Secret capped at 1 MiB. The Altinity, Job and FoundationDB CRDs each inline the full upstream PodTemplateSpec schema, about 600 KB apiece; once the default Strategy CRs render as well, the stored release no longer fits and the forced upgrade that creates them fails with "Secret ... is invalid: data: Too long". The Kafka, RabbitMQ and Redis strategies already carry their pod template schemaless for this reason. Give the three older fields the same markers. The Go types stay typed, the drivers decode the template as before, and the apiserver still validates the pod when the driver creates the workload; every description is kept, and the largest revision drops to about a quarter of the limit. The rationale moves to the package doc, so the six field docs only tell the reader what schemaless means for them. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
aed7fa896e |
docs(release): add changelog for v1.6.5 (#4940)
This PR adds the changelog for release `v1.6.5`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.5.md`.
|
||
|
Myasnikov Daniil
|
4814ccfed9 |
docs(release): note the PostgreSQL restart on upgrade in the v1.6.5 changelog
Clusters with a memory limit below 512Mi and no explicit shared_buffers get the new default on upgrade, which needs a PostgreSQL restart. The changelog described the fix but not the restart. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
cozystack-ci[bot]
|
13f612d55e |
docs: add changelog for v1.6.5
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Myasnikov Daniil
|
c2dab12a2e |
fix(cluster-api): resync capi-operator every minute
A restarted capi-operator v0.19.0 drops the create events for the provider Deployments it finds already running (kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready condition went False while the operator was down therefore keeps it until that Deployment changes or the informer resyncs, and the first resync after a restart lands two periods later because the handler registers after the informer has listed. With the 10m default that is about 20 minutes, during which capi-providers-core times out on the CoreProvider and everything depending on it waits; every further operator restart starts the clock again. With a one-minute period the same repro recovered 2m02s after the operator took its lease, against no recovery within 4m26s with the default on the same cluster. A resync replays the cache and, with providers in sync, ends in "No changes detected" and an unchanged condition, so it writes nothing to the API server; the operator logged about 15 lines a minute and used 2m CPU and 26Mi of memory. The upstream chart has no value for the flag, so a package patch that make update reapplies adds one, and a bats test holds the rendered argument. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
512ffa7ccd |
ci(release): reject truncated entries and docs/next links
Two defects in the v1.7.0 draft were mechanical and passed the validator anyway. Three entries end in an ellipsis right before their attribution: the PR text was pasted up to a length limit, and the tail it lost is where PR authors put what the operator has to do, so the entry is wrong rather than short. And Feature Highlights linked cozystack.io/docs/next/, which the live site answers with 404. Neither appears in any shipped changelog, so rejecting them costs no good output. The link check is anchored on the host because v1.3.0 names the website's content/en/docs/next/ tree in prose, and the ellipsis check only fires before an attribution because v1.4.0 has one inside a sentence and every compare link has three dots. The offending line numbers go into the one-line reason, since the workflows quote the reason into an annotation that ends at the first newline. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e2ca1b7bc5 |
ci(release): overlay the release checks the changelog agent runs
The changelog job checks out the rc tag detached, so every file the agent reads by its working-tree path is the rc tag's copy. Only the agent instructions were refreshed from the dispatch ref. The instructions now tell the agent to run the checks in docs/release.md and hack/validate-changelog.sh on its own output before it exits, and the rc tag's copies of those predate any check added since, or for an old enough tag do not exist at all. Refresh both alongside the instructions. docs/release.md is under neither sparse-checkout path and reaches the tooling checkout only because cone mode also carries the files of every parent directory of docs/agents. The contract test pins both copies and that dependency, so a switch to non-cone patterns fails here rather than as a hard failure of the refresh step at promote time. The AI step keeps its read-only token and its tool flags. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
6395ca1759 |
docs(release): check changelog claims against the code
The generator writes each entry from the opening of the PR text, one PR at a time, and nothing in its instructions asks it to check a statement against the tree it ships. The v1.7.0 draft had every author and PR number right and still carried 22 wrong or missing statements, several in the breaking-changes section, where a wrong instruction breaks an upgrade: a companion-repository fix credited to a tag that does not contain it, worker pools said to survive a migration that never pins their template, migrations said to run on a variant that disables them. The checks that would have caught them live in docs/release.md, which the model is never pointed at, and the detailed rules it does read are about bookkeeping. Add the semantic half: read the whole release note, describe the final state at the upper-bound tag, compare with the previous release, take identifiers, migration behaviour and variant scope from the code, collect required actions beyond `!` commits, pin companion changes to a tag, and run the pre-publish checks and the validator on the written file before exiting. The pre-publish checks name the file after $NEW, which in the promotion configuration is the rc tag, so they are told to read the stable file name; otherwise the reference and link checks read a file that does not exist and report nothing. A minor release's versioned docs are not live until it publishes, so those links are checked against the website tree rather than expected to answer 200. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
8bd6d91dbb |
fix(backups): gate every default Strategy CR on its CRD (#4929)
## What this PR does Upgrade from v1.6.x to v1.7 leaves `backupstrategy-controller` HelmRelease in `UpgradeFailed`: `no matches for kind "Kafka" in version "strategy.backups.cozystack.io/v1alpha1"`, same for Rabbitmq and Redis, and none of the new strategy CRDs get created. CRDs of this chart are ordinary templates, and Helm maps every object of a release through the RESTMapper before applying any of them. Default Strategy CRs for kafka, rabbitmq and redis were gated only on the bucket name, so they render next to their not yet served CRDs, Build() fails, nothing is applied and every retry renders the same manifest. Same happens on a first install with external S3 (`provisionBucket: false`) for all older default strategies, bucket name comes from values there. MongoDB default strategy already has the `crdEstablished` gate, this PR adds it to all other default strategies. On an installed cluster nothing changes. Otherwise first revision creates the CRDs and skips the CRs, then DefaultObjectsGate forces the follow-up upgrade that renders them. CRDs stay templates so later CRD changes still reach the cluster on upgrade. Short explanation is added to `docs/operations/backup-classes.md`. To keep this from coming back there is `hack/backupstrategy-crd-gate.bats`: it renders the chart with the bucket gate open and no CRD served and fails on any Strategy CR from any template, so a new `strategy-*-default.yaml` without the gate goes red without anyone updating a list. It also checks that every CR gates on the CRD this chart ships for its kind. helm-unittest covers both halves of the gate for all 11 templates. Checked on a throwaway kind cluster: - v1.6.4 installed, then upgraded: current main fails with the error above, this branch passes with all 11 CRDs Established and the follow-up upgrade creates the rest - modified CRD is applied on a later upgrade - fresh install converges, and fresh install with external S3 converges here while main fails with "no matches for kind" for eight kinds Fixes #4925. Needs backport to release-1.7, both commits apply there cleanly. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): backupstrategy-controller release no longer gets stuck in UpgradeFailed on upgrade from v1.6 and no longer fails a fresh install with external S3, strategy CRDs are created first and default strategies follow on the next reconcile ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Default backup strategies are now created only after their required resource types are available and the backup bucket is resolved. This helps prevent installation and upgrade issues, including when using externally managed S3 storage. * **Documentation** * Clarified when default backup strategies are created during installation and upgrades. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
f47bab7280 |
test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only checks the templates a case lists, so a new strategy-*-default.yaml that forgets the gate stays green until an upgrade wedges. Render the chart with the bucket gate open and no CRD served, and fail on any strategy.backups.cozystack.io object from any template. Also check that every rendered Strategy CR gates on the CRD this chart ships for its kind, because a gate naming the wrong CRD passes the first check and keeps the CR out of every live render. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7204992452 |
fix(backups): gate every default Strategy CR on its CRD
The chart ships its strategy CRDs as ordinary templates, and Helm maps every document of a release through the RESTMapper before it applies any. A default Strategy CR gated only on the bucket name therefore fails Build() with "no matches for kind" whenever the bucket name resolves before the CRD is served, the release applies nothing, the CRDs are never created, and every retry renders the same manifest. That happens on the upgrade that first ships a kind: the Kafka, Rabbitmq and Redis CRs leave a v1.6 cluster's HelmRelease stuck in UpgradeFailed on the way to v1.7. It also happens on the first render of an external-S3 install (provisionBucket=false), where the bucket name comes from values and every older default CR renders next to its absent CRD. MongoDB already gates on crdEstablished for this reason. Give every default Strategy the same gate. On an installed cluster the CRDs are established and the render is unchanged; otherwise the first revision creates the CRDs and skips the CRs, and DefaultObjectsGate, which reads the cozy-default BackupClass routes, forces the follow-up upgrade that renders them once the kinds are served. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
11b19635e0 |
fix(monitoring): send alerts to every Alertmanager replica (#4889)
## What this PR does vmalert sends every alert to every Alertmanager replica instead of to the shared Service. The chart gave vmalert a single notifier, `http://vmalertmanager-alertmanager.<namespace>.svc:9093`. That Service is headless and resolves to the three Alertmanager pods, but each alert reached one replica at a time and moved to another one whenever vmalert reconnected. The Alertmanager README asks for the opposite. It says "Do not load balance traffic between Prometheus and its Alertmanagers" and to point the sender at every instance, because the cluster deduplicates notifications only when each replica receives each alert. What this looked like on three clusters running 1.6.4, read through the Alertmanager API of each pod. - On all three, every active alert sat on one replica and the other two had none. The counts were 0, 0 and 10 on the first, 0, 0 and 23 on the second, and 1, 0 and 9 on the third. - On the third one, which also has an email receiver, the active alert count of each replica changed 163 to 180 times in 24 hours. The three replicas sent 63, 67 and 62 emails in that window. Over the same week only three alert names with a severity routed to email fired there. An alert that leaves a replica expires there and goes out as resolved, and the replica that receives it next sends it as a new firing notification. So a single long-running alert keeps producing firing and resolved notifications, and a receiver that rejects concurrent duplicates sees them as conflicts. The change. - `vm/vmalert.yaml` renders one notifier per Alertmanager pod, `http://vmalertmanager-alertmanager-<i>.vmalertmanager-alertmanager.<namespace>.svc:9093`. That is the address the operator builds for StatefulSet pods (`VMAlertmanager.asPodFQDN`), and the replicas already reach each other through the same per-pod names in their `--cluster.peer` arguments. - The replica count comes from a new `monitoring.alertmanager.replicas` helper, which now also sets `spec.replicaCount` of the VMAlertmanager. The notifier list cannot drift from the number of pods. Why not the operator's notifier selector. `VMAlertNotifierSpec.selector` expands matched VMAlertmanagers into the same per-pod addresses, but only when the VMAlert is reconciled. The VMAlert controller does not watch VMAlertmanager objects, so a VMAlert reconciled before its VMAlertmanager exists, which can happen on a fresh install, would get no notifier until something else triggers a reconcile. A static list renders the same addresses without depending on that order. Upgrade impact. The VMAlertmanager is unchanged, it renders `replicaCount: 3` as before. The VMAlert gets new notifier arguments, so the operator restarts the vmalert pod once. Testing. - `helm unittest` passes for the monitoring chart, 24 suites and 304 tests. The new suite `tests/vmalert_notifiers_test.yaml` checks the three notifier addresses, the replica count of the VMAlertmanager and the release namespace in the addresses. - Each of these breakages turns at least one case of the new suite red. The notifier back on the Service address, the helper set to 2, a hardcoded namespace in the address, and a hardcoded `replicaCount` that no longer follows the helper. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map in `docs/agents/contributing.md` against the diff. It changes two templates and the helpers of `packages/system/monitoring` and adds a test file. No values, schema, README, package name, make target or `ApplicationDefinition` changes, so no entry applies. ### Release note ```release-note fix(monitoring): vmalert now sends alerts to every Alertmanager replica instead of the shared Service, so the Alertmanager cluster deduplicates them and a long-running alert no longer produces repeated firing and resolved notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * vmalert now configures a notifier URL for each of the three Alertmanager replicas, rather than using a single shared URL. * Alertmanager’s replica count is managed consistently with the notifier configuration. * **Tests** * Added checks for the replica count, all three notifier URLs, and the configured namespace. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
f3486f4e3a |
docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed them and left in the source tree, where no operator looks. They are on the website, so a second copy here only drifts: edits kept landing in docs/ after the website copy was made. docs/vm-external-vlan.md is split rather than moved. Its section on staging the reference CNI plugins is a contract that hack/cni-plugins-staging-contract.bats asserts against the multus Dockerfile, the daemonset manifest and the platform values, so that half is package documentation and goes to packages/system/multus/README.md, which the suite now reads. The operator half goes to the website. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
acf02ef5cf |
docs(agents): define where a new document belongs
The `docs/` tree had accumulated component guides that belong on the website: standalone OIDC, GPU/vGPU and VM-networking documents written as part of a feature PR and never routed anywhere. Nothing said where documentation should go, so `docs/` was the default, and nothing said that the website imports a package README only for the packages on its Makefile lists, so prose written for any other package reached no reader. The destination table turns on one test: whether a non-documentation file has to link to the document by path. A merged document records current behaviour and its limitations rather than the development process around it, because a claim about a phase or an unmerged change turns false once the work ships and nothing re-checks it. backup-classes.md still described the `image.tag` pin as one that an in-flight change removes. The pin has since been removed, so the note now says that. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
e05c56524e |
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ``` |
||
|
Aleksei Sviridkin
|
588d8292e7 |
fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#4819)
## What this PR does Large objects served through a Bucket's S3 endpoint stop tens of megabytes into the download (cozystack/cozystack#3782). The cause is not TLS: the SeaweedFS S3 gateway is OOM-killed mid-response. Its read-ahead for a sequential GET is meant to stay four chunks in front of the reader, but SeaweedFS counts only newly started chunk downloads toward that window and re-runs the prefetch after every 256 KiB read, so the gateway pulls up to about a gigabyte of the object into memory and exceeds the 512Mi limit the S3 deployment gets from its resources preset. The client sees `unexpected eof` over HTTP/1.1 and `INTERNAL_ERROR` over HTTP/2, which is what the CDI importer reported. Upgrading does not help yet: 4.31, 4.45 and the latest release 4.48 all carry the defect. The defect is reported upstream in seaweedfs/seaweedfs#11672, and the fix with its regression test is merged there in seaweedfs/seaweedfs#11674, after 4.48. Until a SeaweedFS release carrying it is vendored, this PR builds SeaweedFS in-tree the way #3343 builds cilium-operator: `images/seaweedfs` downloads the 4.45 source archive and checks it against a pinned checksum, applies `patches/reader-cache-prefetch-window.diff`, runs the tests of the packages the patch touches (including the regression test the patch adds) once, compiles `weed` with the Go toolchain upstream's own builder uses for the tag, and lays it over the upstream `chrislusf/seaweedfs:4.45` image pinned by digest, so everything else in the image stays the upstream build. `weed version` reports `cozystack-patched` in the commit field. All SeaweedFS components run the in-tree image. `make update` re-pins the Dockerfile from the vendored chart: the version, the source checksum, the upstream image digest and the builder toolchain. It fails when the patch no longer applies, which is how the first release carrying the upstream fix will show that the patch is ready to drop. Renovate is disabled for that Dockerfile, since it would move those pins one at a time, and whitespace checks are off for the patch. Tests: - `tests/image_test.yaml` pins every component (master, volume, filer, S3) to the in-tree image with a digest. - `make test-image-version` fails when the Dockerfile's version drifts from the vendored chart's appVersion. - `make update-image`, which `make update` runs, was checked to be idempotent on the current chart, to restore deliberately corrupted pins, and to fail with a clear message when the patch context no longer matches. - `hack/build-matrix_test.bats` now expects seaweedfs to be a build unit next to objectstorage-controller. - The Go regression test inside the patch fails on unpatched 4.45 (read-ahead reaches 60 chunks past the reader) and passes with the patch; the image build runs it. Verified on a test cluster against an 840 MiB object, with every SeaweedFS component running the image this package builds and the S3 gateway at the production limits (512Mi, 1 CPU). Unpatched 4.31, 4.45 and 4.48 gateways were OOM-killed on every full GET. The patched gateway served nine full GETs (six sequential over HTTP/1.1 and HTTP/2, three in parallel) with matching checksums and no restarts; it read from the volume servers about as many bytes as it sent, its live heap after a GET was about 40 MiB, and three parallel GETs peaked at about 140 MiB. The image this PR pins reports `4.45 cozystack-patched` and was rerun through the same checks. Drop the patch, and the in-tree build with it, once a SeaweedFS release carrying the upstream fix is vendored. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): stop the S3 gateway from being OOM-killed while serving large objects, which cut downloads (including CDI imports from a Bucket URL) off tens of megabytes in. SeaweedFS is now built in-tree with a patch that bounds the gateway's read-ahead. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * SeaweedFS now runs a Cozystack-built image based on version 4.45, pinned to a specific image digest. * **Bug Fixes** * Sequential small reads now keep read-ahead within the configured prefetch window, preventing it from extending unexpectedly. * **Build & Maintenance** * SeaweedFS image updates are aligned with the chart version and verify the source archive, builder image, and patches before building. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
58b4343e5a |
ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).
The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
9cc3a9e459 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
Andrey Kolkov
|
9af1c0f902 |
fix(backups): prepare standalone VMDisk restore targets before the data mover runs
collectUnderlyingResources returned nothing for a non-VMInstance kind, so prepareForRestore did nothing for a standalone VMDisk: no HelmRelease suspend and no DataVolume delete. An in-place VMDisk restore then raced Flux/CDI — the restored HelmRelease reinstalled and CDI recreated the DataVolume over the same PVC name the data mover was repopulating, wedging the RestoreJob. Report a standalone VMDisk's own disk as its underlying resource, so the existing (VMInstance-proven) restore preparation suspends the vm-disk-<name> HelmRelease, deletes its DataVolume and — with keepOriginalPVC — renames the live PVC before the data mover recreates and repopulates it. No VM, so no IP/MAC is collected. A VMInstance is halted before its disks are touched; a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk a running VM mounts leaves the original Terminating behind pvc-protection, Velero skips a PVC that still exists, and the RestoreJob would report Succeeded having moved nothing while the VM loses its disk on the next restart. An in-place VMDisk restore now fails before changing anything when a Pod that has not terminated mounts the disk's PVC, naming the Pod and telling the user to stop the VM or restore the VMInstance instead. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so its disk is now derived from the application name and such a restore is prepared and checked for a mounting Pod as well. The per-disk orLabelSelectors stay a VMInstance concern. A VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore that carries both labelSelector and orLabelSelectors, so appending them for a VMDisk would break a strategy written with labelSelector. When an in-place restore fails after preparation, nothing is rolled back: resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the -orig- PVC. The failure message and a Warning event now name the suspended HelmReleases and the -orig- PVCs so the user can recover them, read back from the cluster, so a failure before a step ran does not name what it would have left. Preparation takes the target apart before Velero validates the Restore, so it is refused up front when the Velero Backup is gone or not Completed. That check now runs on every pass behind the same failure path as the rest of preparation, and a FailedValidation Restore fails the RestoreJob like a Failed one instead of leaving it Running. A HelmRelease suspend or a VM halt that did not happen now stops preparation instead of renaming and deleting under a running release or VM, and a VMInstance's disks are renamed only once its launcher Pod has let go of them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name. Velero then skips the PVC as already present, the RestoreJob reports Succeeded and the VM boots a blank disk. Once the DataVolume is gone, preparation now deletes a PVC found under the original name, read uncached, before the Velero Restore is created. Preparation spans several reconciles, so the RestoreJob records a PreparationStarted condition before it first changes the target, and retries without changing anything when recording it fails. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of claiming that nothing changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. Now that a VMDisk target is prepared, keepOriginalPVC=false reaches it too: its DataVolume is deleted with orphan propagation and the bound PVC is deleted before the Restore, as for a VMInstance disk. Left in place, Velero would skip it and CDI would refuse the DataVolume the resumed HelmRelease creates. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Alexey Artamonov
|
e8635f6e4b |
build(seaweedfs): stamp the re-pinned in-tree image
The builder, the compile flags and the version stamp changed, so the image built from them has a new digest, and values.yaml has to name it for the components to run that build. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
ecdc043e63 |
build(seaweedfs): keep Renovate and git whitespace off the in-tree build
Renovate would bump the builder, the base image or their digests one at a time into a combination upstream never shipped, and the next `make update` would silently put them back, so it leaves that Dockerfile to the update recipe. The patches are unified diffs of Go sources, whose context lines start with a marker before tabs and blank lines, which git's whitespace checks would otherwise flag or rewrite. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
5592a3f65d |
build(seaweedfs): re-pin the in-tree build from make update
The in-tree build pinned its source, base image and toolchain by hand, and nothing moved them when `make update` vendored a new chart, so a bump would build one release's weed into another release's image, or keep patching a release that already carries the fix. Pin the source archive by checksum and the upstream image by digest as build arguments, compile with the toolchain upstream's own builder uses for the tag, and let `make update` re-pin all of them from the vendored chart and fail when the patches no longer apply. The tests of the patched packages now run once rather than once per target platform, and `weed version` names the build. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
d2c6b85cac |
fix(seaweedfs): run the in-tree SeaweedFS image in every component
With the image keys unset the vendored chart renders chrislusf/seaweedfs at its appVersion, so the patched build from images/seaweedfs would be built and never run, and the S3 gateway would keep being OOM-killed on large GETs. Point every component at the in-tree image, pinned by digest like every first-party reference, and hold that in a unit test so a values change cannot quietly fall back to the upstream binary. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
1913104bd3 |
build(seaweedfs): build SeaweedFS in-tree with a bounded read-ahead
The S3 gateway is OOM-killed while serving a large object. SeaweedFS starts read-ahead downloads after every read of a sequential stream and counts only the downloads it newly starts toward its prefetch window, so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to about a gigabyte of the object into memory, past the 512Mi limit of the S3 deployment. No SeaweedFS release carries a fix yet, so there is no release to move to instead. Build weed from the 4.45 sources with a patch that keeps the window at the next prefetchCount chunks, and lay it over the upstream image of the same tag so the rest of the image stays upstream's own. The image build runs the tests of the packages the patch touches, including the regression test the patch adds, because go build never compiles them. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |