mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
1551
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
782e0ab64d |
fix(e2e): validate the leak allowlist against every suite layout
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the selectors already see step-file suites and a Test nested below the suite directory. A suite in one of those layouts could not be allowlisted at all: the run stopped before the first suite. Validate the allowlist with the rule hack/select-e2e.sh enumerates suites with, spelled the same way on both sides and cross-referenced so they move together. Describe the three layouts once in the e2e-chainsaw README, point the agent doc at it, and list kubernetes-proxmox as the parked suite. Closes #4659 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
61238500dd |
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it
run-proxmox.sh wrote the tenant's super-admin kubeconfig to $TMPDIR with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. dump_diagnostics then read whatever sat at that path, so a run that failed before the fetch reported the previous run's tenant as its own. The file now lives under COZY_PROXMOX_WORKDIR, which the workflow points at runner.temp. It is written under umask 077, removed at the start of apply-tenant so this run never sees a leftover, removed again in the suite's finally, and once more in the workflow's credential cleanup step for a run killed before finally runs. Closes #4806 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
cac152048e |
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard
cozy_guard_all_helmreleases reads the parent and the kubernetes-<test>- addons. The worker pool is kubernetes-nodes-<test>-md0, an Application HelmRelease the apiserver stamps with RetryOnFailure like the parent, on the node-join path through the talos-reconcile Job it renders. Its name does not start with the addon prefix, so a failed action on it was never read, on the passing path or in the failure-path report. Read it on both paths. The pool name is derived from the parent, the way the addon prefix is, so the two cannot drift. The failure-path report checks that the release exists first: a run that failed between the parent and the pool apply has none, and that is a fact about where the run stopped rather than a failed read. Closes #4737 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
98d0402096 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml sets the same strategy for the same reason and had no pin. Add the mirror test and update the guard comment that pointed at the gap. No template change. Closes #4731 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
7ad7d39349 |
test(hack): keep every chart's stored Helm release within budget
A chart can grow past the 1 MiB Helm release Secret without any test noticing: it renders, lints and installs, and only the upgrade whose manifest is large enough fails, on a live cluster, with a release that stays failed. backupstrategy-controller sat under 1 KB from the limit before it crossed it. Estimate the stored size the way Helm encodes it (release JSON with the chart's templates and files, gzip, base64) and fail any chart under packages/ above 90 percent of the limit, plus the largest revision of backupstrategy-controller with every default Strategy rendered. The estimate counts files a .helmignore names, because the production path stores them. A size measured without its manifest is a lower bound that passes for the wrong reason, so every chart must render offline except an explicit list of charts that need cluster state, and the worst-case render must succeed and contain the default strategies before it is measured. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
513b5088a8 |
test(backups): fail on a strategy CRD that inlines a pod schema
The release-size budget only reports that the backupstrategy-controller release grew too large, not which field did it, and three strategies once inlined the upstream pod schema before anyone noticed. Check the generated strategy CRDs for an inlined PodSpec and name the property path, with the markers that fix it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c2dab12a2e |
fix(cluster-api): resync capi-operator every minute
A restarted capi-operator v0.19.0 drops the create events for the provider Deployments it finds already running (kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready condition went False while the operator was down therefore keeps it until that Deployment changes or the informer resyncs, and the first resync after a restart lands two periods later because the handler registers after the informer has listed. With the 10m default that is about 20 minutes, during which capi-providers-core times out on the CoreProvider and everything depending on it waits; every further operator restart starts the clock again. With a one-minute period the same repro recovered 2m02s after the operator took its lease, against no recovery within 4m26s with the default on the same cluster. A resync replays the cache and, with providers in sync, ends in "No changes detected" and an unchanged condition, so it writes nothing to the API server; the operator logged about 15 lines a minute and used 2m CPU and 26Mi of memory. The upstream chart has no value for the flag, so a package patch that make update reapplies adds one, and a bats test holds the rendered argument. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
512ffa7ccd |
ci(release): reject truncated entries and docs/next links
Two defects in the v1.7.0 draft were mechanical and passed the validator anyway. Three entries end in an ellipsis right before their attribution: the PR text was pasted up to a length limit, and the tail it lost is where PR authors put what the operator has to do, so the entry is wrong rather than short. And Feature Highlights linked cozystack.io/docs/next/, which the live site answers with 404. Neither appears in any shipped changelog, so rejecting them costs no good output. The link check is anchored on the host because v1.3.0 names the website's content/en/docs/next/ tree in prose, and the ellipsis check only fires before an attribution because v1.4.0 has one inside a sentence and every compare link has three dots. The offending line numbers go into the one-line reason, since the workflows quote the reason into an annotation that ends at the first newline. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e2ca1b7bc5 |
ci(release): overlay the release checks the changelog agent runs
The changelog job checks out the rc tag detached, so every file the agent reads by its working-tree path is the rc tag's copy. Only the agent instructions were refreshed from the dispatch ref. The instructions now tell the agent to run the checks in docs/release.md and hack/validate-changelog.sh on its own output before it exits, and the rc tag's copies of those predate any check added since, or for an old enough tag do not exist at all. Refresh both alongside the instructions. docs/release.md is under neither sparse-checkout path and reaches the tooling checkout only because cone mode also carries the files of every parent directory of docs/agents. The contract test pins both copies and that dependency, so a switch to non-cone patterns fails here rather than as a hard failure of the refresh step at promote time. The AI step keeps its read-only token and its tool flags. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
fe053ce93a |
fix(kubernetes-nodes): hold factory-CA downloads to https redirects
With COZY_TALOS_FACTORY_CA set, curl -L still followed a redirect to plain http, so the image could arrive without the TLS verification the CA was given for. The CA path now passes --proto-redir =https, in line with the refusal of a CA for an http factory. The no-CA program is unchanged. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
000479a0cc |
feat(kubernetes-nodes): trust a factory CA in the Proxmox template script
A pool can now point CDI at a self-hosted Image Factory served by an internal CA through talos.imageFactoryCA. hack/proxmox-talos-template.sh downloads the same image with curl on the hypervisor, which trusts only the system store and fails there with the same unknown-authority error. COZY_TALOS_FACTORY_CA names a PEM file with those certificates. The program writes them after the hypervisor's system bundle and hands the result to curl --cacert, so they are trusted on top of the system store, as CDI does, and the hypervisor's own trust store is not changed. The file is pasted into the root program, so it must be complete certificate blocks and nothing else; a private key sharing the file, trailing text or an unclosed block is refused, as is a CA given for a plain-http factory. Without the variable the program is byte-for-byte what it was. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
f47bab7280 |
test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only checks the templates a case lists, so a new strategy-*-default.yaml that forgets the gate stays green until an upgrade wedges. Render the chart with the bucket gate open and no CRD served, and fail on any strategy.backups.cozystack.io object from any template. Also check that every rendered Strategy CR gates on the CRD this chart ships for its kind, because a gate naming the wrong CRD passes the first check and keeps the CR out of every live render. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f3486f4e3a |
docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed them and left in the source tree, where no operator looks. They are on the website, so a second copy here only drifts: edits kept landing in docs/ after the website copy was made. docs/vm-external-vlan.md is split rather than moved. Its section on staging the reference CNI plugins is a contract that hack/cni-plugins-staging-contract.bats asserts against the multus Dockerfile, the daemonset manifest and the platform values, so that half is package documentation and goes to packages/system/multus/README.md, which the suite now reads. The operator half goes to the website. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e05c56524e |
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ``` |
||
|
|
58b4343e5a |
ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).
The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
|
9cc3a9e459 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
|
1913104bd3 |
build(seaweedfs): build SeaweedFS in-tree with a bounded read-ahead
The S3 gateway is OOM-killed while serving a large object. SeaweedFS starts read-ahead downloads after every read of a sequential stream and counts only the downloads it newly starts toward its prefetch window, so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to about a gigabyte of the object into memory, past the 512Mi limit of the S3 deployment. No SeaweedFS release carries a fix yet, so there is no release to move to instead. Build weed from the 4.45 sources with a patch that keeps the window at the next prefetchCount chunks, and lay it over the upstream image of the same tag so the rest of the image stays upstream's own. The image build runs the tests of the packages the patch touches, including the regression test the patch adds, because go build never compiles them. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7011510456 |
fix(fluxcd): keep the hook waiting on a release it cannot read
The pre-delete hook read a release's state once, so a single failed read was taken as the release being handled: its fence came off before helm-controller's uninstall, and the report counted it as released cleanly. A state read is now retried within the release's share of the window, a release whose state still cannot be read keeps its fence for the wait like one helm-controller has not handled, and the end of the wait reads each again before naming what the release was left to. A release still suspended after the sweep is named as suspended but not by flux-plunger, which also holds when no manager is recorded for the field at all. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
699555c7cb |
fix(fluxcd): give each release its own share of the hook's retries
The pre-delete hook's retry budget was one window spent in listing order, so a release that could never be read or unfenced, listed first, took all of it and every release after it got a single attempt. The same transient failure then released a fence or not depending on where its release sorted. Each release now gets its own share of the window, capped at the window's end, and the steps that cover every release keep the whole of it. Two comments said more than holds. The retry window is measured from the script's start, not the Job's, so scheduling and the image pull come out of its one-minute reserve rather than being covered by it. And the five-minute conflict retry inside kubectl scale never runs here, since without a precondition it sends a merge patch with no resourceVersion to conflict on. helm-controller's finalizer, which the hook reads to tell whether a release being deleted is still held, is now tied to the constant the controller uses, as the fence already was. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
af50c4a6b4 |
fix(fluxcd): let helm-controller uninstall before the hook unfences
The pre-delete hook lifted flux-plunger's suspension of a release being deleted and removed its fence in the same step. helm-controller runs the uninstall of a release being deleted once it is no longer suspended, but when its own finalizer is already gone the fence is all that keeps the object, so removing the fence at once deleted the release before helm-controller could reconcile it, without an uninstall. The hook now keeps such a fence while helm-controller handles the generation the lifted suspension wrote, until shortly before its retry window closes, and names a release it still had to let go of as before. A release being deleted under a suspension another manager also holds was reported as cleanly released, although helm-controller skips the uninstall of a suspended release; it is now named as deleted without one and counted with the others. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
43b02bda30 |
test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app kind and the chart prefix the helpers derive sentinel and Secret names from. The demo BackupClass routes Valkey to the same demo strategy. The valkey suite gains valkey-backup-roundtrip, which drives that demo end to end: seed a marker, back it up, restore it in place and into a copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a seaweedfs change now selects the valkey suite too, and since the round-trip runs examples/backups/redis as its harness, an edit there selects both the redis and the valkey suites. The reduced E2E install keeps the backup stack for the valkey suite, as it does for the other backup round-trip suites. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
|
0c2f131192 |
feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and speaks the same replication protocol and RDB format, so it needs no strategy kind of its own. The only difference the backup Job sees is the chart's resource-name prefix: valkey-<app> instead of redis-<app> for the sentinel Service and the auth Secret. The Redis driver now accepts applicationRef kind Valkey as well as Redis and hands the template .Release.Prefix, derived from the app's kind, which cozy-default-redis uses for every name it builds. A kind with no known prefix fails the render instead of producing an empty one. cozy-default routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must target an app of the same kind as the backup's source: replaying a Redis dump into Valkey, or the reverse, is not something the strategy can promise across engine releases. The shipped template reads the prefix as `.Release.Prefix | default "redis-"`: during an upgrade the previous controller, which has no Prefix in its render context and serves only Redis, renders the new CR, and a bare reference would print "<no value>" into the names. The redis:8 image already loads RDB v11, which both shipped Valkey versions (7.2, 8.1) write. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
|
cfa9c5a5c7 |
fix(e2e): keep chainsaw asserts polling through transient API errors (#4867)
## What this PR does Chainsaw asserts in the e2e suites can fail long before their timeout. Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other API error. When etcd changes its leader, kube-apiserver answers the GET with a 500, and whichever suite happens to be polling goes red. Between 2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached chainsaw, which is 6 of the 23 assert failures in that window. Each one has `etcdserver: leader changed` in the apiserver logs within 0.2 s of the failure. Five of the six came after 10-09, and four of them were the computeplane ingress-nginx assert. The fix is kyverno/chainsaw#2761, which is still open and not in any release. Until it is released, the e2e sandbox image builds chainsaw v0.2.15 from source with that PR's commit applied unchanged, using the same Go minor and build flags as the v0.2.15 release binary. The build runs the tests of the patched packages. It also runs chainsaw against a fake apiserver that answers 500: a short run of 500s must not fail the assert, and an endless one must fail it at its timeout. Plain v0.2.15 fails both cases after the first GET. The Dockerfile header lists what to remove once a chainsaw release carries the fix. The README sentence added here has to go too, and so does the Renovate rule that keeps the builder on Go 1.26. The upstream PR also breaks one chainsaw test outside the packages it touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks the new `; operation terminated with error: context deadline exceeded` suffix. I found it with a full `go test ./...` on v0.2.15 with and without the patch. The image build tests only the patched packages, so that test does not run here. One trade-off: when the API stays broken, an assert or `error` step now runs to its timeout instead of failing within seconds, so a red run caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw installed on its runner, not the sandbox image, so it does not get this fix. Known gaps, left for later: the new test covers only an assert on a named object, not one that lists objects or an `error` operation, and its skip guard does not notice if no test runs at all. The patch file has no `-whitespace` entry in `.gitattributes`, so `git diff --check` reports whitespace inside the upstream patch. A multi-platform build of the image would run the builder tests once per platform; the image is built for amd64 only. Closes #4720 ### Screenshots Not a UI change. ### Downstream repositories None of them builds or uses the e2e sandbox image; the change stays inside this repo's CI. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end sandbox assertions now retry after transient Kubernetes API errors instead of failing immediately. Persistent errors still cause assertions to fail when their timeout expires. * **Testing** * Added checks that verify assertions recover after temporary API errors and fail within the expected timeout when errors persist. * **Maintenance** * Updated sandbox image build and dependency update settings for the Chainsaw testing tool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d060c1dbb0 |
fix(kafka): pass the container limits to the Kafka JVMs (#4865)
## What this PR does The Kafka JVMs we ship size themselves from host memory instead of their container limits, and get OOMKilled. The Strimzi 0.45.1-rc1 images carry JDK 17.0.15, which reads `/proc/cgroups` to decide whether it runs in a container. Kernels 6.12 and newer built without `CONFIG_MEMCG_V1` no longer list the memory controller there, so the JVM turns container support off. The JDK fix is JDK-8347811, backported to 17.0.20 and 21.0.10. In CI this shows up in two places. The cluster operator gets a 25 GiB max heap inside its 512Mi limit. In 107 of the last 150 e2e reports the kernel OOM-killed a Java process at about 500M, the limit the cluster operator shares with the topic and user operators. Brokers and controllers start with a 5 GiB heap inside a 1Gi limit, because Strimzi asks the same JDK for the container memory and takes half of what it reports. This PR passes the limits to every Kafka JVM explicitly: - the cluster operator gets `-XX:MaxRAM` and `-XX:ActiveProcessorCount` from its own limits through the downward API; - brokers, controllers, the topic and user operators and the drain-time Cruise Control get `JAVA_TOOL_OPTIONS` with `-XX:MaxRAM` set to their memory limit, plus `-XX:ActiveProcessorCount` where the chart sets a CPU limit. The chart sets one only for brokers and controllers. A LimitRange default on the entity operators or Cruise Control is invisible to the chart, and Strimzi template env takes no `resourceFieldRef`, so those JVMs still count host CPUs. Every java in a container reads `JAVA_TOOL_OPTIONS`, including the heap probe Strimzi runs before starting Kafka. So Strimzi's own heap rules and Kafka's G1 defaults apply to the real memory limit, and the chart does no heap arithmetic. `jvmOptions` with `-XX` would not work here: Strimzi turns those into `KAFKA_JVM_PERFORMANCE_OPTS`, and Kafka skips its G1 settings when that is set. Each JVM now prints a "Picked up JAVA_TOOL_OPTIONS" line on start. In a kind cluster on such a kernel the operator then runs Serial GC with a heap of about 100 MiB, and peaks at 249Mi while creating a Kafka, against 440Mi before. A broker with a 1Gi limit and one CPU gets the same `-Xmx518982205` that Strimzi computes when the JDK sees the limit, with G1 and one concurrent GC thread. Existing Kafka clusters roll once on upgrade, Talos ones included, because the pod environment changes. Their heap does not change on a kernel where the JDK already sees the limit. The chart has no values key for JVM options or heap, so there is no user setting this could override. The Kafka migration hook now applies with `--openapi-patch=false`. Its container runs under a tenant LimitRange default of 128Mi, and a plain `kubectl apply` of an existing pool downloads the cluster's whole OpenAPI document, about 300Mi with the platform's CRDs, so every upgrade in such a tenant failed on an OOM-killed hook. With the flag kubectl builds the same merge patch from the last-applied annotation and peaks at about 35Mi. A larger memory limit would count against the tenant quota, and a server-side apply would stop removing fields that `kubectl create` set on the `kafka` pool. Other Jobs without resources are tracked in #4868. The operator OOMKill is what triggered the e2e failure in #4855. The other half is a Strimzi race. After a restart, the PodSet controller drops the ADD event of a PodSet that has no pods yet and only picks it up at the 5-minute resync. That one is upstream and is described in the issue. The other JVMs we ship are tracked in #4864. Talos nodes are not affected, the Talos kernel keeps `CONFIG_MEMCG_V1=y`. Generic-variant clusters on newer distribution kernels are. CI is affected even though it runs Talos, because there the Talos nodes are containers on the runner's Ubuntu 6.17 kernel. Closes #4855 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kafka): the Strimzi cluster operator, Kafka brokers and controllers, the entity operators and the drain-time Cruise Control no longer size their JVM heap from host memory on kernels built without cgroup v1 memory support, which got them OOMKilled. The Kafka migration hook no longer runs out of memory on upgrade in a tenant with resource quotas. Existing Kafka clusters roll once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Kafka components now size JVM memory and processor settings based on their configured resource limits, including during migrations. - The Kafka cluster operator now receives container resource limits for JVM sizing. - **Bug Fixes** - Kafka migration operations avoid OpenAPI patch lookups that could use excessive memory. - **Tests** - Added coverage for JVM sizing, resource limits, and migration behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
785cbc391b |
fix(fluxcd): keep Helm uninstall working during flux-plunger recovery (#4293)
## What this PR does Fixes #4060. `flux-plunger` recovers a HelmRelease stuck with `has no deployed releases` by briefly suspending it, deleting the stale Helm storage Secret, recording a processed-version annotation, and unsuspending. helm-controller v1.5.x skips the Helm uninstall and drops its finalizer when it handles a delete on a suspended HelmRelease, and it wakes on the same delete event flux-plunger does, so a delete arriving inside that window orphaned every resource the release owns (for a VMDisk: DataVolume, PVCs, the CDI importer pod, RBAC, WorkloadMonitor and the storage Secret itself). Lifting the suspension on the delete event cannot win that race reliably. The bug is not specific to VMDisk. The change makes such a delete reach the uninstall: - **An uninstall fence.** The suspension is applied together with a flux-plunger finalizer, `flux-plunger.cozystack.io/uninstall-fence`, in one server-side apply, so flux-plunger never holds a suspension without it. If a delete arrives while the release is suspended, the fence keeps the object whether helm-controller or flux-plunger handles the delete first: flux-plunger lifts its suspension and keeps the fence, helm-controller reconciles the unsuspended generation and runs the uninstall, and flux-plunger removes the fence once helm-controller's finalizer is gone and it has observed that generation. A recovery that completes lifts the suspension and the fence together. A terminating HelmRelease never begins recovery, and termination is re-checked immediately before deleting the storage Secret. - **Only flux-plunger's own suspension is acted on.** Ownership is decided by which field manager owns `spec.suspend` in `metadata.managedFields`: the apiserver transfers the field away the moment an operator (or `cozyhr resume`, or a backup/restore controller) writes a different value to it, so a later suspension is never misread as flux-plunger's. A suspension set through the shared `flux-client-side-apply` manager (platform migrations, the tenant delete hook, an operator's manual `kubectl apply`) is never claimed. A release deleted under another manager's suspension is that manager's decision: flux-plunger does not hold it with the fence, and helm-controller leaves its resources in place, as the migrations that suspend and then delete rely on. - **No suspension taken over in a race.** Acquiring applies without forcing first and, on a conflict, re-fetches and forces only if `spec.suspend` is still `false`: a backup controller owns the field at `false` via a dynamic-client `Update`, which a non-forced apply can never take. Every apply flux-plunger sends, acquiring as well as lifting the suspension or the fence, is preconditioned on the `resourceVersion` and `uid` of the read that decided on it. The `resourceVersion` makes a write landing in between fail the apply instead of being absorbed: acquire then defers rather than adopt or take over a suspension set meanwhile, and lifting the suspension and the fence from a release read as live cannot delete it without an uninstall if helm-controller has handled a delete in the meantime. The `uid` stops an apply to a release that has gone from creating an empty one, since server-side apply creates what does not exist. A conflict is retried through the requeue. The apply carries only `spec.suspend` and the fence, so it cannot leak the required `spec.interval` and conflict on it. - **Fresh reads.** The release, every re-check and the storage Secret list are read from the apiserver (an uncached `APIReader`), not the informer cache. A lagging cache would let the processed-version guard delete the revision below the one just reclaimed, and the ClusterRole grants secrets `get`/`list` but not `watch`, which an informer needs to stay synced; flux-plunger deliberately does not cache Secrets. Every step requeues with backoff on failure. - **Events.** A Warning when flux-plunger holds a suspension it failed to clear, and a Warning when a delete has been held by the fence for more than ten minutes waiting for helm-controller (the held state is re-checked on a timer, not only on helm-controller's next status write). A release deleted under another manager's suspension gets a Normal event when that is a migration's choice, and a Warning when the release carries flux-plunger's processed-version annotation, because then the skipped uninstall is an accident, not a decision. Normal events also mark an externally-owned suspension left alone and a recovery deferred because the release changed between read and write; an external suspension on a release flux-plunger has recovered before is a Warning instead, because it is most likely one an earlier flux-plunger left mid-recovery (see Operating notes). Events from the shutdown drain are written straight to the apiserver, since the manager's recorder stops with the manager. - **The fence is released when flux-plunger goes.** Once its manager has stopped, and with it every reconcile, flux-plunger drains its fences under a deadline inside the pod's termination grace period; a stop that overran its 10s bound is drained too and still reported: a recovery in progress is aborted, the suspension and the fence lifted together for the next flux-plunger to start again, and a release being deleted has flux-plunger's suspension lifted and keeps the fence until helm-controller has run its uninstall. A release helm-controller does not handle in time keeps its fence. The drain lives in the controller, so the envtest suites hold it against a real apiserver's generations and finalizers. A pre-delete hook on the chart is the backstop for what the drain could not finish, so uninstalling the component never leaves a release it cannot finish deleting: it stops flux-plunger, waits out its grace period, then lifts any remaining suspension and fence without waiting again, naming each release it lets go of before helm-controller uninstalled it. It lifts a suspension only where flux-plunger alone holds it, the outcome the controller's own relinquish has, with a JSON patch of `spec.suspend` preconditioned on the `resourceVersion` and `uid` it read; not with `kubectl apply --server-side`, which on a release last written by a client-side `kubectl apply` moves that apply's fields to the applying manager and strips every one the body leaves out. helm-controller runs a pre-delete hook under the release's uninstall timeout, 300s by default and not set by the operator, so the Job's deadline sits under it. A Deployment that cannot be stopped or a listing that never succeeds ends the hook before any fence is touched, and a fence it cannot remove ends it after the sweep; either way it exits non-zero, because a failed pre-delete hook leaves the release untouched and helm-controller retries the uninstall, hook included, whereas exiting 0 would let the component go and strand the fence. A flux-plunger pod still there after its grace period, typically one stuck `Terminating` on a `NotReady` node, is only reported, and the pod wait is cut to what is left of the hook's retry window; because such a pod may still be setting fences, the fences are listed again after the sweep and any found fails the hook. The Job runs once and is kept after success so its report stays readable until its TTL; `helm.sh/hook-output-log-policy` is deliberately not used, because helm-controller never sets Helm's hook output writer and calls a nil one for it (fluxcd/helm-controller#1562). The hook's kubectl image is digest-pinned in the chart values, and its RBAC is only what `kubectl scale`, the listing and the patches issue. The chart's README documents the finalizer and how to clear it by hand. A single writer is kept by leader election, not by the rollout. A second process acting under the same field manager could clear a suspension the first is still recovering, and no Deployment strategy prevents one: the next pod starts as soon as the old one is terminating (Kubernetes v1.35 has no pod-replacement policy for Deployments), and an eviction or a node drain replaces a pod the same way, while the old one stops and drains. The chart now passes `--leader-elect` with a Role on `leases` in the release namespace. The Lease is not given up on stop and outlasts the last renewal, the renew deadline, the manager's bounded stop (10s) and the drain, and only a flux-plunger that led drains at all, so the next one acts only once the old one is done; the price is that a new pod waits for the Lease to run out, a little under two minutes at the default. The timings derive from one value, `shutdownGracePeriodSeconds` (90 by default, refused outside 40 to 140 at render): the pod's termination grace period, the drain deadline (25s less: the manager's 10s stop plus a 15s margin before the kubelet's SIGKILL, both tied to the Go constants by a contract test) and the hook's pod wait. The strategy keeps `maxSurge: 0` and `maxUnavailable: 1` and stays `RollingUpdate` rather than `Recreate` on purpose: the chart shipped without a strategy block, so every existing install's live Deployment carries the apiserver-defaulted `rollingUpdate` object and helm-controller applies server-side, where laying `type: Recreate` over that populated block is rejected on upgrade. ### Operating notes A HelmRelease carries the fence only while flux-plunger is recovering it or waiting for helm-controller to uninstall it. A fenced release whose uninstall keeps failing stays `Terminating` exactly as any HelmRelease does under helm-controller's own finalizer, and raises the Warning above. Stopping flux-plunger drains its fences, and uninstalling it releases whatever the drain left through the pre-delete hook; a fenced release that is deleted while flux-plunger is merely scaled to zero, or killed without a graceful stop, and that helm-controller has not uninstalled, stays `Terminating` until flux-plunger runs again. To release one by hand, after confirming its uninstall has run or is not wanted: ```bash kubectl patch helmrelease NAME --namespace NAMESPACE --type json --patch '[{"op":"test","path":"/metadata/finalizers/INDEX","value":"flux-plunger.cozystack.io/uninstall-fence"},{"op":"remove","path":"/metadata/finalizers/INDEX"}]' ``` where `INDEX` is the position of `flux-plunger.cozystack.io/uninstall-fence` in `metadata.finalizers`; the `test` operation makes the patch fail rather than remove another finalizer if the list has changed. Earlier flux-plunger releases set `spec.suspend` through the shared `flux-client-side-apply` manager, so a suspension one of them left behind cannot be told apart from a migration's or an operator's and is treated as external: it is not cleared automatically. Such a leftover exists only if the old pod stopped in the middle of a recovery, and flux-plunger raises a `SuspensionNotOwned` Warning on it. The chart README carries this section too. To list candidates, suspended releases carrying the flux-plunger annotation whose `spec.suspend` is not owned by flux-plunger's apply: ```bash kubectl get helmreleases.helm.toolkit.fluxcd.io --all-namespaces --output json | jq --raw-output '.items[] | select(.spec.suspend == true) | select(.metadata.annotations["flux-plunger.cozystack.io/last-processed-version"] != null) | select([.metadata.managedFields[]? | select(.manager == "flux-plunger" and .operation == "Apply" and .fieldsV1["f:spec"]["f:suspend"] != null)] | length == 0) | "\(.metadata.namespace)/\(.metadata.name)"' ``` Clearing one is an operator action: confirm nothing else (a migration, a restore) is holding it, then resume it with `flux resume helmrelease NAME --namespace NAMESPACE`. ### Testing The regression the issue asks for is `TestEnvtest_DeleteDuringRecoveryReachesUninstall`: against a real apiserver, a delete lands while flux-plunger holds its recovery suspension, and helm-controller's v1.5.0 delete handling is reproduced step for step (uninstall only when not suspended, finalizer dropped either way, observed generation recorded). It covers both orders, helm-controller handling the delete first and flux-plunger handling it first; without the fence the first order leaves the release gone with no uninstall. The same suite pins the managedFields ownership dynamics a fake client cannot represent: the field transfer, the forced acquire over a dynamic-`Update` owner, the `resourceVersion` precondition rejecting a write that races either apply, the boundary that a `flux-client-side-apply` suspension is never claimed, and interval preservation. The fence mechanics were also checked against a live helm-controller: with another finalizer holding a deleted, suspended release, lifting the suspension made helm-controller run the uninstall and remove the release's resources. The envtest suite runs on every pull request: a new `make test-controllers-envtest` target builds a pinned `setup-envtest` and fetches a kube-apiserver and etcd into `_out/envtest`, which CI caches and the target reuses without asking the release index, takes the HelmRelease CRD from the in-tree Flux manifests, and CI calls it next to `make test-controllers`; the control plane gets a longer start timeout because it starts beside the other test goals. The drain is held there too: it aborts a live recovery, carries a delete through the uninstall and lifts the fence only once helm-controller has observed the unsuspended generation, keeps the fence when helm-controller never answers, releases a fence left without a suspension at once, and leaves another manager's suspension alone. `TestEnvtest_TheNextFluxPlungerLeadsOnlyAfterTheDrain` runs two real managers on one Lease: a clean stop still drains, and the second manager becomes leader only after the first one's drain has ended; it goes red if the Lease is given up on stop. Unit tests cover the controller logic (every branch, ordering, requeue, and the ownership decision through an injected seam), the Lease timing relations, that only a leader drains and through the manager's API reader rather than its stopped cache, that a failed listing is retried, that a stop which overran its grace period is still drained and its events reach the apiserver, and contract tests tie `main.go`, the chart's flags, the drain's arithmetic and the hook's names to the Go code. The pre-delete hook is covered by helm-unittest and by `hack/flux-plunger-fence-hook_test.bats`, which runs its rendered script against a fake `kubectl` that keeps each release's finalizers, suspension owner, generation and observed generation, bumps the generation on a spec write as the apiserver does, answers NotFound without `--ignore-not-found`, spends simulated time on waits, lists a release once however many copies of the fence it carries, refuses `kubectl apply`, lifts a suspension only through the preconditioned JSON patch and never one another manager also holds, reads a negative `kubectl wait` timeout as a week as kubectl does, and applies a finalizer removal only when it is the guarded test-then-remove at an index that holds the fence. Every mutation in a set covering each predicate conjunct, each retry, each exit path, the re-listing, the timings and the selector turns the hook suite red, as does every mutation in the Go set covering the drain's branches, the Lease timings and the events; three drain mutations that still survive are tracked in #4782. The hook script was also run as a Job in the hook's own image on a live cluster, succeeding, with the Deployment already gone, and with patch denied by RBAC, where it exited non-zero and named the release it left; that run caught that a per-call `--request-timeout` makes kubectl drop its in-cluster config, so the hook sets none and the fake rejects it. A later live run caught `kubectl apply --server-side` stripping the finalizers and spec of a release created by a client-side apply; with the JSON patch the same releases kept every field, the suspension flux-plunger alone held was lifted, a co-owned and a foreign one stayed, and every fence went. ### Remaining limitations - A delete that arrives after recovery has deleted the storage Secret of a release whose only revision was that one finds no revision to uninstall; helm-controller then treats the release as already gone. - A writer that sets `spec.suspend: true` while flux-plunger already holds it at `true` does not take the field over, because the value does not change. If such a writer then deletes the release inside the recovery window (the suspend-then-delete that the tenant delete hook and some platform migrations use to skip the uninstall), flux-plunger still sees its own suspension, lifts it, and helm-controller runs the uninstall that writer meant to skip. The same held before this change, when the end of recovery unsuspended unconditionally. - The already-processed guard compares revision numbers only. When Helm re-creates a revision under the same number after recovery deleted it (a release whose only revision fails again the same way), recovery runs again for it. - `listHelmReleaseSecrets` reads the local cluster only, so a HelmRelease that targets a remote cluster via `spec.kubeConfig` (whose storage Secrets live in the remote cluster) is not recovered; it is skipped, never mis-handled. - A new flux-plunger pod waits for the old one's Lease to run out before it acts, on every restart and after a crash: a little under two minutes at the default. Recovery is not latency-sensitive, and this is what keeps a single writer while the old pod drains. - The leader Lease is left in the release namespace after the chart is uninstalled, as with any controller-runtime leader election; it is inert. ### Screenshots Not applicable: no UI change. ### Downstream repositories The trigger map was walked against the diff: it touches `internal/controller/fluxplunger`, `cmd/flux-plunger`, the `packages/system/flux-plunger` chart (templates including RBAC for the leader Lease, a pre-delete hook, values, tests, its package `Makefile`, README), a new bats test under `hack/`, the root `Makefile` (a new `test-controllers-envtest` target; existing targets are unchanged) and the pull-request workflow that calls it and caches its binaries. No row matches: ccp gates on existing `hack/` files and make targets, and nothing there is moved, renamed or changed in behaviour; a new test file and a new target reach none of them. The website's FluxPlunger section describes the recovery steps, which still hold; it does not describe the fence or the drain, which the chart README now documents. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(fluxcd): a HelmRelease deleted while flux-plunger is recovering it now reaches the Helm uninstall instead of leaving its resources (DataVolume, PVCs, RBAC, storage Secret) behind ``` |
||
|
|
e05cef4d48 |
fix(ci): dispatch frozen-line builds with GITHUB_TOKEN
The CI app lacks Actions write access, so freezing a new release line leaves its initial packages build undispatched. Use the repository token for workflow_dispatch without expanding the app installation permissions. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-by: LLM |
||
|
|
7c46ffff12 |
fix(backupstrategy-controller): run backup clients on a package-owned image (#4756)
## What this PR does The Altinity (ClickHouse) and RabbitMQ backup strategy Pods ran `platform-migrations`: a 349 MB image carrying `kubectl`, `helm`, `git`, `cozyhr` and `etcd-migrate`, whose `ENTRYPOINT` is `run-migrations.sh`. Both pins were stamped from `packages/core/platform/Makefile`, so any migration change re-digested the backup clients too, and only the `command:` override kept the Pods from running platform migrations (#3420). - New `packages/system/backupstrategy-controller/images/backup-http-client/`: `alpine:3.24` (digest-pinned, the same base the tree already uses) + `curl` + `jq` + `ca-certificates`, `USER 65532:65532`, no `ENTRYPOINT`. Everything is installed at build time, so the Pods still never run `apk add` and keep working air-gapped. - New `image-backup-http-client` target in the package's own `Makefile`, chained after `image-redis-backup` like the existing ones so only one `yq -i` writes `values.yaml` at a time. It stamps `chBackupClientImage` and `rabbitmqBackupClientImage` with the same ref. - Removed the cross-package stamps from `packages/core/platform/Makefile`, the WORKAROUND/TODO comments in the chart and the RabbitMQ example, and the stale "live violation" line in `docs/agents/image-refs.md`. - `hack/image-pin-consistency.bats`: dropped the `platform-migrations`-specific test, whose subject no longer exists. The general check (no repository pinned at more than one digest, computed from `hack/promote-retag.sh --dry-run`) still covers the two keys sharing one image. **Why one image for both strategies rather than `ch-backup-client` as the issue named it.** The RabbitMQ strategy had the same reuse and the same cross-package stamp; leaving it would have kept the stamp alive. Its script needs `curl --aws-sigv4` + `ca-certificates`, a subset of the ClickHouse one, so one image serves both. The two values keys stay separate so either strategy can move to its own image later. **Initial pin.** The image does not exist in ghcr until CI builds it, so both keys are committed as `backup-http-client:v1.6.0` without a digest, the same way `redis-backup` was introduced. PR CI rebuilds touched packages and overlays the fresh refs for e2e; release preparation's `make image` stamps tag+digest. Image size: 6.9 MB vs 349 MB. Verified locally: - `docker buildx build` for `linux/amd64,linux/arm64`; in the image as uid 65532: curl 8.22 with `--aws-sigv4`, jq 1.8.2, `date`, `seq`, writes to `/tmp` (the RabbitMQ script's only write), TLS verification against the bundled CA store. - `helm unittest` (23/23), `helm template` renders both strategies with `backup-http-client` and no `platform-migrations` reference. - `hack/cozytest.sh hack/image-pin-consistency.bats` and `hack/build-matrix_test.bats` pass. Not yet run end to end on a cluster; the e2e backup round-trips will exercise it. Fixes #3420 ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map against the diff: it touches `packages/system/backupstrategy-controller` (new image, Makefile target, values comments/pins, template comments), `packages/core/platform/Makefile` (stamp removal only, no `values.yaml` key change), `hack/image-pin-consistency.bats`, `docs/agents/image-refs.md` and a comment in `examples/backups/rabbitmq`. No app package, schema, CRD, variant, asset, platform values key or `hack/package.mk` change. The website does not list first-party image repositories (searched for `platform-migrations` and `redis-backup`), so no mirror list needs the new one. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(backupstrategy-controller): the ClickHouse (Altinity) and RabbitMQ backup strategy Pods now run a dedicated `backup-http-client` image (alpine + curl + jq + ca-certificates, non-root, ~7 MB) instead of the 349 MB `platform-migrations` image. Air-gapped mirrors need the new `ghcr.io/cozystack/cozystack/backup-http-client` repository. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Altinity and RabbitMQ backups now use a dedicated backup HTTP client image, configured separately from the platform migrations image. * The client image includes the tools required for backup operations and runs with a non-root identity. * Both backup configurations receive the same digest-pinned image reference when the image is built. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ebceef4105 |
ci(retention): sweep packages named by the published nightly tree (#4826)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The Artifact Retention workflow has failed every day since mid-August, and it never pruned anything. It lists the org's container packages with `GET /orgs/cozystack/packages?package_type=container`, and that endpoint answers `400 Invalid argument` to the app installation token the job uses. Others hit the same in actions/github-script#449 and cli/cli#9606. An old `|| true` used to hide it. This is item A19 in #3278. The per-package endpoints accept the same token: a probe run confirmed it, and the dry-run below read every package with it. So the job no longer lists the org. The new `hack/retention-packages.sh` takes the package names from the published `cozystack-packages:nightly` tree, reading refs through the shared `hack/lib/image-refs.sh`. It also adds `cozy-installer` and `cozystack-nocloud`, which the nightly publishes outside the tree. The committed tree can't be the source: it misses images whose digest only the build pins, and those still get dated nightly tags every day. Failures now turn the run red instead of passing silently: - if the name list is empty, the nightly sweep is skipped and the run fails, but abandoned promotion candidates are still handled, because their protection doesn't depend on that list - if a package's versions can't be read, that package is skipped and the run fails at the end - if a DELETE is refused, `hack/retention-delete.sh` logs an error with the HTTP status, goes on with the remaining versions and fails the run (before, this was only a warning) - if names resolve but none of the packages is linked to this repository, nothing gets swept, so the run fails too, after the promotion sweep has run (that sweep itself is skipped only when candidate protection can't be resolved) Promotion-candidate protection itself is unchanged. The contract test that pinned the old org listing now pins the new behaviour, and both new scripts have bats tests. Known gap: a component that disappears from the nightly tree drops off the list, so its old dated nightlies are never pruned. Today none of the packages outside the list has more dated nightlies than the keep window. Dry-run on this branch is green: https://github.com/cozystack/cozystack/actions/runs/38075159900. It resolves 59 packages, skips `cozystack/cozystack-ui` with the warning, and would delete 418 old nightly versions plus one abandoned promotion candidate. Deleting needs a different token. The app token lists versions, but a one-version apply run (https://github.com/cozystack/cozystack/actions/runs/37939905343) got `404 Package not found` on DELETE for a version it had just listed. The same DELETE with this repository's `GITHUB_TOKEN` and the job's `packages: write` worked in a second run (https://github.com/cozystack/cozystack/actions/runs/37941084456), with no change to package settings. So `hack/retention-delete.sh` now deletes with `GITHUB_TOKEN`, and listing stays on the app token. That delete was shown on a package linked to this repository. One of the swept packages, `cozystack/cozystack-ui`, is linked to another repository, and I didn't try deleting there. Such a package is skipped before its versions are read and named in a warning, so it can't turn every apply run red. The follow-up for it is in #3278. `RETENTION_APPLY` stays off, so the scheduled run is a green dry-run until someone turns it on. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> N/A ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(retention): the GHCR retention workflow takes package names from the published nightly tree instead of the org package listing, which rejects the app token, and fails the run whenever names, versions or deletions cannot be resolved. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Nightly package retention now targets packages referenced by the published nightly tree and keeps the 14 newest eligible versions per package. * Promotion candidates older than 30 days are eligible for pruning only when their digests are not protected by qualifying release pull requests or branch tips. * **Bug Fixes** * Packages linked to another repository are skipped, and sweep failures are reported without preventing other eligible packages from being processed. * **Documentation** * Clarified which nightly versions and promotion candidates qualify for retention. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
34488db766 |
ci(e2e): never tear down the Proxmox tenant on the runner's default context (#4807)
## What this PR does The Proxmox e2e teardown could delete from the wrong cluster. It runs under `always()` and got its kubeconfig only from `GITHUB_ENV`, which the "Stage credentials" step writes. If a step before that fails, for example a missing stand variable, `KUBECONFIG` is never set. The three `kubectl delete` calls then go to whatever cluster the self-hosted runner's own kubeconfig selects. Every teardown call now passes the staged file with `--kubeconfig`. Stage fails unless that file's current context resolves to a server, and prints kubectl's own error when it does not. Before, an empty `COZY_KUBECONFIG` secret let client-go fall back to its defaults, in-cluster credentials among them. The suite step gets the staged path in its own `env`. The `GITHUB_ENV` export is gone, so no step in this workflow picks its cluster up implicitly. Tear down decides whether to skip from how Stage ended, not from the file. If Stage did not succeed, the suite never ran and there is no tenant to remove, so the step prints a notice and exits 0. If Stage succeeded and the file no longer resolves, the step fails with kubectl's error, because the tenant and its VMs may still be there. A side effect: a run whose Stage fails no longer cleans up a tenant left by an earlier `keep_tenant` run. The next run that passes Stage without `keep_tenant` removes it. The Stage and Tear down scripts now have unit tests in `hack/proxmox-teardown-contract.bats`, run against a stub `kubectl`. I also ran both against a real kubectl v1.37.1 for every Stage outcome. The files were missing, empty, garbage YAML, without a current context, with a dangling context, without a server, and valid. One more case was a runner without kubectl. The deletes also no longer stop at the first failure. Under `set -e`, a failed first delete skipped the tenant delete, and the tenant's VMs stayed on the hypervisor. Now each delete runs, and the step fails at the end if any of them failed. With a recording `kubectl` that fails one chosen delete, all three ran in every case and the step exited 1. With the change reverted, a failing first delete was the only one that ran. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): the Proxmox e2e teardown always uses the staged kubeconfig, so it can no longer delete from the runner's default cluster. It skips only when Stage did not succeed and fails when the kubeconfig breaks after Stage. A failed delete no longer skips the tenant delete. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Improved reliability of automated Proxmox test setup and cleanup. Tests now stop setup when Kubernetes configuration is unavailable, skip cleanup when setup fails, and attempt all resource deletions before reporting cleanup failures. * **Tests** * Added checks for configuration validation, cleanup behavior, and handling of failed deletion attempts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
2db18a0c86 |
test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport (#4657)
## What this PR does The kubernetes suites' LINSTOR cleanup check now compares dataset names per node, and prints the leftover names when it fails (#4651). This PR puts the rest of what a triager needs into `cozyreport.tgz` on every run, green or red: the size of each dataset, its `origin` and `creation`, the snapshots, the pool's pending frees, and the LINSTOR volume list. What `hack/cozyreport.sh` now collects: - `linstor/zfs-pools.txt`: the zpool behind each ZFS storage pool, per node, taken from `StorDriver/StorPoolName` in `linstor --machine-readable sp l`. - `linstor/zfs/<node>.txt`: that node's satellite runs `zpool get -p freeing,leaked,allocated,free` and `zfs list -p -t all -o name,used,avail,refer,origin,creation -r` over the node's pools. - `linstor/volumes.txt`: `linstor v l`, next to the existing `resources.txt`. Sizes are exact bytes, so the sizes of leftover datasets can be summed against the pool root's `avail`. That `avail` can differ from `storage-pools.txt` without anything leaking: on a node with only thick pools, `linstor sp l` serves the free capacity cached at the satellite's last report. A dataset LINSTOR could not destroy because snapshots depend on it is renamed to `_deleted_<name>` in the same pool, so a search for leftovers has to look for that prefix as well as `pvc-`. A pool that `zpool get` or `zfs list` cannot open does not cost the listing of the other pools. `zpool get`'s message and exit status go into the listing, and `zfs list` lists the rest. A node LINSTOR names no ZFS pool for gets a note instead of a listing, because `zfs list -r` with no pool argument lists every pool on the machine. A satellite whose `spec.nodeName` cannot be read gets a note that says so. All reads go through the bounded exec reader, like the other LINSTOR reads, and the time budget in the script header counts them. The second commit changes how that reader marks a read that wrote output and then exited non-zero without hitting the timeout. It used to append `TRUNCATED ... part way through, not because the command finished`. But `kubectl exec` passes the remote command's status through, and `zfs list` over one missing pool prints a complete listing of the others and exits 1. The marker now names the exit status and says the status can't tell an error after complete output from a cut stream. This changes the marker for every LINSTOR exec read. The timeout marker stays as it was. Tested with `hack/cozytest.sh hack/cozyreport.bats`. The new tests run the controller and satellite shells against stub `linstor`, `zpool` and `zfs`, with the real `jq`. They cover a failed and an unparsable storage-pool read, a pool row with a missing, empty or blank backing name, a non-ZFS pool, a pool name with a space, a pool that `zpool` and `zfs` cannot open, a failing `zfs`, and an unreadable node name. A test goes red if any of these break: the filter that keeps non-ZFS pools out, the filter that drops a missing, empty or blank pool name, the jq exit status, the `zpool` fallthrough, the node-name note, or the reader's new wording. Refs #4653 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(e2e): cozyreport records the ZFS datasets of each node's LINSTOR pools in exact bytes, and the LINSTOR volume list, on every e2e run. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Collection reports now include LINSTOR volumes, ZFS pool capacity and usage, and per-node ZFS datasets and snapshots with exact-byte details. Dataset listings are scoped to the ZFS pools LINSTOR reports for each node. * Reports include notes or warnings when node, pool, or dataset information is missing or cannot be collected. * **Documentation** * Clarified storage report contents, artifact markers, bounded reads, and how unsupported data, incomplete results, collection limits, and per-object warnings are reported. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
647407545d |
fix(backupstrategy-controller): run backup clients on a package-owned image
The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the migration runner, and whose pins were stamped from packages/core/platform so every migration change moved the backup clients too. Build images/backup-http-client (alpine + curl + jq + ca-certificates, non-root) from this package's Makefile and stamp both client keys from it. The cross-package stamp in packages/core/platform/Makefile and the platform-migrations-specific check in hack/image-pin-consistency.bats go away; the general one-digest-per-repository check still covers the two keys sharing one image. Fixes #3420 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
33ca8b67bb |
feat(redis): run Sentinel on the same image as the Redis pods (#4831)
## What this PR does
The Redis application pins its Sentinels to `redis:6.2.6-alpine`
(`packages/apps/redis/templates/redisfailover.yaml`), while the data
pods run the version selected by `version` (`redis:8.4.0` for `v8`,
`redis:7.4.7` for `v7`). The pin was added in
|
||
|
|
9a348eeb50 |
fix(kubernetes): delete orphaned resources in new post-delete job (#3856)
## What this PR does When tenant deletes the kubernetes cluster, 3 resources are orphaned and leftover in the tenant namespace. - jobs/<name>-oidc-bootstrap - secrets/<name>-talos-ca - secrets/<name>-talos-tls-cert This PR adds a new "step 7" in existing `packages/apps/kubernetes/templates/delete.yaml` in order to delete these orphaned resources manually. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): Fixed the `<name>-talos-ca` and `<name>-talos-tls-cert` Secrets and the `<name>-oidc-bootstrap` Job being left in the tenant namespace after a Kubernetes cluster is deleted. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Kubernetes release cleanup now removes the Talos CA and TLS certificates before their associated Secrets, and also removes the OIDC bootstrap Job. * The final cleanup report now includes best-effort failures from Step 7, alongside failures from Steps 3, 5, and 6. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
be55f6eb2b |
test(ci): load the shared bats helper in the labeler contract test
Every unit bats file under hack/ must load test_helper so it runs with nounset, and hack/bats-strict-setup.bats fails the suite for a file that does not. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
79e95a1b30 |
test(ci): pin the label-writing workflows with contract tests
pr-labeler.yaml and stale.yaml decide which labels land on issues and PRs, and neither had a test behind it. A mistyped label name is not an error, it is a rule that silently stops matching. hack/label-workflows-contract.bats pins: - every label either workflow can write or match resolves against a name: in .github/labels.yml (typeToKind, scopeToArea, the three literal adds, stale/close labels, both exempt lists) - the typeToKind mappings are exactly the rows of the type table in docs/agents/contributing.md, in both directions, and a type the table lists under "(no kind label)" has no key in the workflow - the trigger list stays [opened, reopened, synchronize] and 'edited' does not come back in either YAML spelling - the labeler stays additive-only (addLabels only, no removeLabel or setLabels) with the area/uncategorized fallback - stale policy values: label choices, 12 issue and 4 PR exemptions, days-before-stale 60, days-before-close 14, operations-per-run 100 - stale runs from its daily cron and workflow_dispatch only JS // and YAML # comments are stripped before matching, so a mapping or a gate demoted to a comment cannot satisfy the contract. Negative checks use the grep -c / -eq 0 form of the sibling contract files: a negated pipeline is ignored by set -e, so it cannot fail a test under hack/cozytest.sh and fails under bats only as the last command. Every assertion was mutation-checked under both bats and hack/cozytest.sh. POSIX-only constructs; picked up automatically by BATS_UNIT_FILES. Closes #3589 Signed-off-by: shaurya703 <kesar.shaurya@gmail.com> |
||
|
|
904bc38fc3 |
fix(kubernetes): delete the Talos PKI and oidc-bootstrap Job on uninstall
cert-manager runs with enableCertificateOwnerRef: false, so the <release>-talos-ca and <release>-talos-tls-cert Secrets carry no ownerReference to their Certificates and outlive the release, keeping the CA private key of a cluster that no longer exists. A new cluster created under the same name then adopts that stale CA instead of issuing its own trust root. The <release>-oidc-bootstrap Job is a hook whose only delete policy is before-hook-creation. Hooks are not part of the release manifest, so uninstall never removes it. The pre-delete hook deletes the two Certificates before their Secrets, because deleting a Secret while its Certificate exists makes cert-manager re-issue it. It runs after the TenantControlPlane is gone, since the control plane's CSR signer mounts the CA. The Role grants each delete by resourceName only. Assisted-by: LLM Signed-off-by: Ray Low <crossrt@gmail.com> |
||
|
|
6ba440f7a3 |
fix(kafka): skip the OpenAPI patch lookup in the migration hook
The pre-install/pre-upgrade hook applies the node pools and the drain KafkaRebalance with kubectl, and its container has no resources of its own, so in a tenant with resource quotas it runs under the LimitRange default of 128Mi. When it applies an object that already exists, kubectl downloads the cluster's whole OpenAPI v2 document only to decide that a custom resource takes a JSON merge patch: with the 374 CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade hooks failed". --openapi-patch=false skips that lookup and builds the same three-way merge patch from the last-applied annotation, peaking at about 35Mi on the same cluster. A server-side apply would be as small but changes what gets removed: the "kafka" pool is created with "kubectl create", whose manager keeps co-owning every field it set, so a later server-side apply leaves those fields in place when the chart drops them. A larger memory limit would charge every upgrade against the tenant quota instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9a88afe6ca |
feat(redis): run Sentinel on the same image as the Redis pods
The Sentinels were left on redis:6.2.6-alpine when the operator moved to the freshworks-oss fork, so that its new compiled-in default could not move them on upgrade. 6.2 is end of life, and every instance now runs a Sentinel one or two major versions behind the Redis it monitors. The rollback risk behind that pin, a newer engine rewriting dump.rdb in a format the older one cannot load, does not apply to Sentinel. It stores no RDB or AOF, only sentinel.conf on an emptyDir that the config-copy init container rewrites from the operator's ConfigMap on every start, so moving back to an older image starts from a clean config. Taking the image from the same version map as the data pods also means one image per node instead of two. The e2e comment that relied on busybox's timeout exit code in the Sentinel container is updated, since that container now carries coreutils. Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com> Assisted-by: LLM |
||
|
|
813c618086 |
fix(e2e): keep chainsaw asserts polling through transient API errors
Chainsaw v0.2.15 ends an assert or error operation on the first API error other than NotFound. kube-apiserver answers a GET with 500 when etcd changes its leader, so a leader change that lands inside a poll fails the step long before its own timeout, on whichever suite happens to be polling. The sandbox image now builds v0.2.15 from source with the fix proposed upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw release carries it. The build runs the tests of the patched packages and a test that drives chainsaw built from the patched tree against a fake apiserver: a run of 500s must not fail an assert, and a persistent one must fail it at its timeout rather than at the first error. Closes: #4720 Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6852f723b8 |
fix(kubernetes-nodes): refuse a newline in the Proxmox template tags
COZY_TEMPLATE_TAGS is comma-separated, but the normalisation also split on a newline, so `talos-v1.\n13` became two tags where the operator meant one. A newline there is now refused, like in every other value the program receives. The stub tests clean up their scratch directory after the last assertion, so a failing test leaves its qm, curl and ssh records behind for inspection. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
a8635f4f68 |
fix(kubernetes-nodes): refuse multi-line values in the Proxmox template script
matches() held each value to its shape with grep -x, which matches line by line, so a value with a newline passed when any one of its lines did and carried the others into the program create runs as root. A value with a newline is now refused, for every field the program receives. The test file loads test_helper and spells its absence check without `!`, for the two Bats audits on main. The Talos version check accepts the pre-release suffix the chart's render accepts, and a failure after qm create names the qm destroy that frees the VMID for the next run. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
bbd39c94a4 |
feat(kubernetes-nodes): build the Proxmox worker template from the pool's Talos values
A proxmox pool clones its workers from a VM template the chart never sees, and nothing said which Talos image that template has to carry. The one an operator reaches for, the platform release's nocloud-amd64.raw.xz, is the bare-metal node image: its zfs extension's service waits for a /dev/zfs no VM has, the boot sequence never completes, and Talos reboots every worker each time the boot timeout runs out, about every 70 minutes on v1.13. It also follows the host nodes' Talos version, not the pool's talos.version that the installer in the worker machineconfig names. The chart's default schematic has a trap of its own: its qemu-guest-agent service waits the same way for the agent's virtio port, which Proxmox adds only to a VM with the agent enabled. hack/proxmox-talos-template.sh builds the template from the chart's talos.schematicID and talos.version in the nocloud flavour with the agent enabled, refuses a tag set another template already carries, because capmox then finds two and clones neither, and prints the templateTags that select it. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
22e0fdbc63 |
test(tests): stop unit tests from reaching a cluster (#4804)
## What this PR does One unit test was reading from whatever cluster the caller's kubeconfig selected. The knob test in `hack/run-kubernetes-node-join_test.bats` runs `cozy_report_node_join_failure` in a fresh `bash -c` and stubs only `kubectl`, as a shell function. `cozy_diag_read` runs every read through `timeout`, and `timeout` execs the real `kubectl` from PATH, so the stub never ran. With a recording `kubectl` first on PATH, one run of that file logged 170 calls such as `kubectl -n tenant-test describe virtualmachineinstances.kubevirt.io`. All of them were reads, but a unit suite should not touch a cluster at all. The PR has three parts: 1. Both `bash -c` blocks in that file now stub `timeout`, as the post-source test already did. They also put recording `kubectl` and `talosctl` executables first on the subshell's PATH and fail if either one is executed. The declined-path block made no calls in my runs, because a zero budget declines every read. It gets the same guard because it has the same shape. 2. The unit and controller targets now run with no cluster to reach. The root `Makefile` sets `KUBECONFIG=/dev/null` and an empty `KUBERNETES_SERVICE_HOST` for `unit-tests`, each of its prerequisites, and `test-controllers`, so the CI command and each target run on its own are covered. The second variable is needed because client-go falls back to in-cluster credentials when the kubeconfig is empty and that variable plus a service-account token are present. The new `hack/unit-lane-no-cluster.bats` runs every prerequisite of `unit-tests` as its own goal with a sentinel kubeconfig and service host in the environment, and fails if any target receives them. 3. One unit file was missing the strict-mode helper. `hack/bats-strict-setup.bats` requires a `load test_helper` line in every unit file, and `hack/fork-new-images.bats` is the only one without it, so the bats lane is red on main and would stay red here. #4805 carries the same one-line fix on its own; if it lands first, this commit drops out on rebase. I ran the whole lane (`make unit-tests test-controllers`) with recording fakes for 20 cluster clients first on PATH. It made no `kubectl` call. `helm` was called 444 times, all `template` or `unittest`, none with `--validate`. `flux` was called once, for `push artifact --help`. Deleting either export line from the Makefile, or removing one target from the list, turns the new test red. Removing the `timeout` stubs turns the knob test red, and the failure lists the reads that escaped. Closes #4793 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(tests): unit and controller test targets run with no kubeconfig and no in-cluster service host, so a unit test can no longer reach a cluster. ``` |
||
|
|
2e5fd56954 |
fix(e2e): stop calling a failed exec read after output a cut
kubectl exec passes the remote command's status through, so a non-zero exit after output does not say whether the stream broke or the command finished with an error. zfs list is the common case: it lists every dataset it can open, skips one it cannot and exits 1, so a complete listing got a TRUNCATED marker and a note claiming it held only what was streamed before the exit. The in-file marker now names the exit status and says the status cannot separate the two, and the note beside it says the same. The timeout branch keeps its own wording. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d195a56793 |
test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport
The report carries the LINSTOR resource list and the pool totals, but nothing that names the datasets holding a pool's space, so a leftover zvol, a snapshot, a clone and pool metadata cannot be told apart from the artifact alone. Each satellite now lists every dataset of the ZFS pools LINSTOR records for its node, snapshots included, with the pool's pending frees. Sizes are exact bytes with avail, so the sizes of leftover datasets can be summed against the pool root's avail; the rounded form loses a sub-GiB difference on a large pool. That avail can legitimately differ from `linstor sp l`, which serves a cached free capacity on nodes with only thick pools. The pool names come from LINSTOR rather than from a name pattern. One pool that zpool cannot open does not cost the listing of the others, and a satellite whose node name cannot be read says so instead of pointing at the pool map. The controller's volume list is recorded next to the resource list. Everything runs on every job, so a red listing can be read against a green one, and goes through the bounded exec reader like the other LINSTOR reads. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
da1f1cc387 |
ci(retention): delete with the workflow token
The app installation token lists package versions but its DELETE of a listed version is answered with 404, while the repository's GITHUB_TOKEN with packages: write deleted the same version without any change to package settings. Deletes therefore go through GITHUB_TOKEN; listing keeps the app token. That deletion was shown on a package linked to this repository. A package linked elsewhere is not attempted: it is skipped before its versions are read and named in a warning, so it cannot turn every apply run red. A failed read of the link still fails the run, and so does a sweep in which no resolved package is linked, since that sweep prunes nothing. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
274f3f7023 |
ci(retention): fail the run when a version cannot be deleted
A refused DELETE only logged a warning, so a token without delete rights left an apply run green with nothing removed. Move the delete into hack/retention-delete.sh, which reports each refusal as an error carrying the API status, still tries the remaining versions, and exits non-zero; the workflow turns that into a failed run after the sweep. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
f1c0f6f3f6 |
ci(retention): sweep packages named by the published tree
The org container-package listing answers 400 to the app installation token, so the job has never reached its sweep. Take the names from the published nightly tree via hack/retention-packages.sh and keep calling only the per-package versions endpoints, which accept that token. An unresolved name list now fails the run: the nightly publishes into these packages daily, so an empty list cannot mean there is nothing to prune. It skips only the nightly sweep. The promotion sweep reads one fixed package and is kept safe by the protected set, which the name list does not feed, so it still runs. A versions read that fails, in either sweep, is reported with the API error and fails the run once the rest is swept, instead of aborting the step through set -e. A failed candidate read leaves no ids behind. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
bde3a01699 |
test(hack): pin the Proxmox lane's kubeconfig guards
Runs the Stage and Tear down scripts out of the workflow against a stub kubectl, so a teardown that exits 0 with the tenant still live, stops at the first failed delete, or hides kubectl's error, or a step that reaches a cluster without naming the staged kubeconfig, fails in unit tests instead of on the stand. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |