mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-06 03:57:52 +09:00
0538e7059161f3cd9ad0219829b5ae05aeebd79b
1543
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
ba0b1ac6ef |
fix(e2e): validate the leak allowlist against every suite layout (#4972)
`lg_allowlist_validate` in `hack/e2e-chainsaw/_lib/leak-guard.sh`
accepted only `<suite>/chainsaw-test.y*ml`, while the selectors already
see step-file suites and a Test nested below the suite directory
(#4630). A suite in one of those layouts could not be put on the leak
allowlist at all: validation failed and the run stopped before the first
suite.
- `lg_suite_has_test` validates an allowlist entry with the rule
`hack/select-e2e.sh` enumerates suites with: `chainsaw-test.yaml`,
`chainsaw-test.yml`, or a numbered step file, anywhere in or below the
suite directory, with the selector's type test (a step-file name counts
unless it is a directory, a `chainsaw-test.y*ml` counts whatever its
type, so Chainsaw reports a directory by that name); a parked
`chainsaw-test.yaml.disabled` matches none. Both sides carry a comment
pointing at the other so they move together.
- `hack/e2e-chainsaw/README.md` describes the three layouts once, under
Layout, and lists `kubernetes-proxmox` as the parked suite.
`docs/agents/e2e-testing.md` points at it.
- Unit tests: every layout is accepted, including a `chainsaw-test.yaml`
that is a directory; a parked, empty, or not-a-step-file directory is
refused, including `1a-install.yaml` (passes the `find` glob, not the
step-file regex) and a directory named `01-install.yaml`.
Left as they are, with the reasons:
- `docs/agents/e2e-testing.md` calling `kubernetes-proxmox` the one
parked suite is correct on main now: `hack/e2e-chainsaw/backup/` was
removed in
|
||
|
Aleksei Sviridkin
|
013b166967 |
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard (#4970)
`cozy_guard_all_helmreleases` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the parent `kubernetes-<test>` release and the addons selected by the `kubernetes-<test>-` prefix. The worker pool is `kubernetes-nodes-<test>-md0`, an Application HelmRelease the aggregated apiserver stamps with `RetryOnFailure` like the parent, and it sits on the node-join path through the talos-reconcile Job it renders. Its name does not start with the addon prefix, so a failed action on it was never read, on the passing path or in the failure-path report. This reads it on both paths, with the `required` strategy check the parent gets, since the apiserver stamps both. The pool name is derived from the parent (`cozy_pool_helmrelease_name`), the way the addon prefix is, so the two cannot drift. The failure-path report checks that the release exists before reading it, with `get hr --ignore-not-found -o name`: a run whose pool apply failed has no pool release (exit 0, empty output), and the report says so instead of reporting a failed read. For that run to reach the report at all, `run_kubernetes_test` now arms the exit handler (`_tenant_failure_on_exit`, with `CURRENT_TENANT_PARENT_HR`) right after the parent `Kubernetes` apply instead of after the `KubernetesNodes` apply: before, a failing pool apply exited under errexit with no handler installed, and the parent went unclassified. The snapshot half still no-ops until the tenant kubeconfig is set. A non-zero exit (a timeout, a kill, a Forbidden) is reported as the failed read it is, through `_cozy_guard_read_failure`, never as absence. The pool reads are gated on the report's own deadline as well as the phase clock, like the addon listing. Unit tests in `hack/run-kubernetes-remediation_test.bats` cover a failing pool release on the passing path, a clean one (the composite returns zero), an empty strategy on the pool, the report with and without the pool release, a cut-off existence read, and a pool read past the report budget; the ordering pin now also requires the handler to be armed before the pool apply. The kubectl stub gained an existence read: an `ABSENT` marker prints nothing and exits 0 under `--ignore-not-found` (NotFound without it, as kubectl does), and the failure markers apply to it as `<name>.exists`. Verified with `hack/cozytest.sh` and Bats on `hack/run-kubernetes-remediation_test.bats` (51 tests), `hack/run-kubernetes-node-join_test.bats` and `hack/remediation-guard.bats`. Closes #4737 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Kubernetes release checks now include worker-pool releases alongside the parent release and add-ons, and flag missing or invalid install strategies and remediation failures. * Failure reports distinguish worker-pool releases that are absent from those whose status cannot be read. Existing releases are assessed against the phase gate and report time budget; when the budget is exhausted, skipped checks are identified. * Failure handling now begins immediately after the parent resource is applied, covering errors that occur while applying the worker pool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Marian Koreniuk
|
b61ad29783 |
fix(e2e): point the selector's suite rule back at the allowlist rule
The first commit says the two rules are cross-referenced so they move together, but only lg_suite_has_test pointed at the selector. The selector's suite enumeration comment now names lg_suite_has_test as well, so a change to either side prompts the other. The layout case also pins a chainsaw-test.yml that is a directory, the other half of the type test: dropping that name from the find now turns the case red. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
75bb70e58d |
fix(e2e): arm the tenant failure handler before the pool apply
The exit handler was armed after the KubernetesNodes apply, so a failing pool apply exited under errexit with no handler installed: the parent went unclassified, and the report's "pool release does not exist" line described a run that could never reach it. Arm it right after the parent Kubernetes apply instead. The snapshot half still no-ops until the tenant kubeconfig is set, so the only change is that a failed pool apply now gets the parent classified and the absent pool reported. The ordering pin now also requires the handler to be armed before the pool apply. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
Aleksei Sviridkin
|
048c029e29 |
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it (#4971)
`hack/e2e-chainsaw/_lib/run-proxmox.sh` wrote the tenant's super-admin kubeconfig to `$TMPDIR/proxmox-e2e-tenant.kubeconfig` with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. `dump_diagnostics` then read whatever sat at that path, so a run that failed before `fetch_tenant_kubeconfig` reported the previous run's tenant as its own. - The file now lives under `RUNNER_TEMP`, which the runner exports into every step and empties between jobs; `COZY_PROXMOX_WORKDIR` overrides it for a local run, and `TMPDIR` is the last fallback. The workflow sets no path of its own: the `runner` context is not available in job-level `env`. - `fetch_tenant_kubeconfig` writes it under `umask 077` in a subshell, since the workflow's `umask 077` covers only the staging step. - `apply-tenant`, the suite's first step, removes a leftover before doing anything else, so diagnostics in this run can only see this run's file. - The suite's `finally` removes it, before the `keep_tenant` exit, and the workflow's "Remove staged credentials" step removes it again for a run killed before `finally`. Verification: `sh -n` and shellcheck on the script; actionlint on `e2e-proxmox.yaml` (only the custom `proxmox` runner label is reported, as on main). `hack/proxmox-e2e-fixture_test.bats` has three new cases with a stub `kubectl`: the fetched file has no group or other bits under `umask 022`, `apply-tenant` removes a planted leftover, and the suite's `finally` (run out of the Chainsaw file on the `keep_tenant` path) removes the file, so `diagnostics` after it prints no tenant section. Each goes red with its line (`umask 077`, the leftover `rm`, the `finally` `rm`) removed. All `hack/proxmox-*.bats` pass (fixture 9, teardown contract 8, step1b 4, talos template 12), also under `hack/cozytest.sh`. Not run on a Proxmox stand: the only stand I have cannot dispatch the workflow at the moment, so the workflow steps themselves are unexercised beyond actionlint. Closes #4806 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Proxmox end-to-end test runs store tenant credentials in a temporary work directory and create kubeconfig files with restricted permissions. * Stale tenant kubeconfig files are cleared before a new tenant fixture is applied. Cleanup removes the file even when tenant data is retained or a run is interrupted, preventing credentials from appearing in later diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
44848b9c9d |
fix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digest (#4917)
## What this PR does
`v1.7.0-rc.1` and `release-1.7` ship the Proxmox
cloud-controller-manager as
`ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.15.0`, a bare
tag. That tag was re-pushed upstream on 2026-09-27: it used to name the
2026-08-06 build (`sha256:fbc553921145…`) and now names
`sha256:d81702998ae1…`, a binary whose gitVersion is `0.15.0` without
the leading `v`. component-base v0.37 parses that string in an `init()`
and panics:
```
panic: version string "0.15.0" doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"
```
The CCM Deployment crash-loops, no new Node loses
`node.cloudprovider.kubernetes.io/uninitialized`, and workloads on new
workers stay Pending. On a live Proxmox cluster this went unnoticed for
five days: the existing Nodes were already initialized, and the failure
only showed when the next new worker came up. Every fresh install pulls
the broken build, so as it stands v1.7.0 would be the first stable
release with the Proxmox substrate, shipping a CCM that cannot start.
Two commits, both cherry-pick onto `release-1.7` unchanged:
1. **`fix(kubernetes)`**: run v0.16.1, pinned as tag plus digest in
`images/proxmox-cloud-controller-manager.tag`. That is the same shape
the chart already uses for the KubeVirt cloud provider. v0.16 builds
with goreleaser and stamps the version with its `v`. The config the init
container writes still parses: the cluster entry moved to
go-proxmox-pool's `ClusterConfig` with the same `url`, `insecure`,
`token_id`, `token_secret` and `region` keys, and `features.provider:
capmox` still writes `proxmox://<SMBIOS UUID>`, the providerID capmox
puts on the Machine.
2. **`fix(kubernetes, proxmox-csi)`**: pin the CSI driver from the same
registry at its current v0.20.0, with no version change. The
management-side controller goes through
`images/proxmox-csi-controller.tag`. The tenant-side node plugin goes
through `repository` + `tag` in
`packages/system/proxmox-csi/values.yaml`, which also puts it in
`hack/lib/image-refs.sh` and so in front of the release multi-arch gate.
Pinning takes the version out of the vendored chart's hands, so
`hack/proxmox-csi-driver-pin_test.bats` now fails unless the chart's
`appVersion` and both pins name the same release.
All three digests are OCI indexes for `linux/amd64` and `linux/arm64`.
The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes through image promotion, which does
not move a tag.
### Testing
- `helm unittest`: `packages/apps/kubernetes` 31 suites / 346 tests,
`packages/apps/kubernetes-nodes` 37 / 223, all passing.
- bats: the new test, plus `image-pin-consistency`,
`promote-rewrite-tags_test`, `overlay-main-images_test`,
`nightly-mirror_test`, `promote-retag_test`, `stitch-multiarch_test`,
`bats-strict-setup`, `bats-runner-coverage`: 107/107. The new test was
checked to fail on a node tag at another version.
- `git cherry-pick -x` of both commits onto `release-1.7` (
|
||
|
Aleksei Sviridkin
|
040ae377ae |
test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool> (#5000)
## What this PR does The Proxmox suite (`hack/e2e-chainsaw/kubernetes-proxmox/`) has now run for the first time on a real self-hosted stand: a dedicated Cozystack v1.7.0-rc.1 management cluster on Proxmox VE, with the runner CageForge offered to host in cozystack/community#98. The suite failed before it reached the hypervisor, because the fixture and the chart disagree on what a node pool may be called. `kubernetes-nodes` names its release `kubernetes-nodes-<CR name>` and, since |
||
|
myasnikovdaniil
|
5c564f9154 |
fix(release): keep release bodies within GitHub's size limit (#5142)
## What this PR does Finalize sets release body after the stable tag, `api/apps/v1alpha1` tag and `release-X.Y` fast-forward already exist. Body that GitHub refuses fails the "Publish draft release" step at that point and leaves release a draft, and rc-to-stable image retag with `cozy-installer` chart publish never run. GitHub's web UI cut the v1.7.0-rc.2 changelog at about 125000 characters, v1.7.0 changelog is 259,026. Release can now carry short hand-written `docs/releasenotes/vX.Y.Z.md` that becomes its body, changelog stays the full record. Finalize and `update-releasenotes.yaml` take release notes if they exist and are not blank, otherwise changelog. Both hold the body to 100000 bytes. Bytes are never fewer than characters and the limit sits 25 percent under the cut seen in web UI. Exact GitHub limit and its answer to an oversized body don't matter. Over the limit finalize publishes one-line link to the changelog on the tag, with a warning in the run log. v1.6.0 changelog is 97,186 bytes, so a next release near that size without release notes gets the link. Sync never writes the link, it warns and leaves published body alone, because `tags.yaml` backstop brings changelog to `main` without release notes and sync would replace them. `update-releasenotes.yaml` also runs on pushes touching `docs/releasenotes/v*.md`. `docs/release.md` describes the rule, including that release notes have to reach `main` too. Release without release notes and with changelog under 100000 bytes is published as before, byte for byte. ## Tests `hack/release-changelog-behaviour.bats` runs real `script:` of both workflow steps under node on fixture trees. Finalize cases are release notes over changelog, blank release notes, changelog only, neither file, 100000 bytes accepted and 100001 replaced by link, and 40000 three-byte characters (120000 bytes) counted as over. It also runs finalize over every shipped changelog, so each one at or under the limit publishes byte for byte. Sync makes the same choice, never updates release whose body would be over the limit, and leaves up-to-date and missing ones alone. `hack/release-changelog-contract.bats` pins new trigger path and the 100000 constant. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(release): a release can carry short docs/releasenotes/vX.Y.Z.md that is published as its GitHub release body, changelog stays the full record. Release body over 100000 bytes is published as a link to the full changelog on the tag, so publishing a release with very long changelog no longer fails after its tags exist, and the release notes sync leaves such a body alone. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Release Management** * Release publication and subsequent updates now prefer non-empty versioned release notes, falling back to the changelog when needed. * If neither source contains content, the existing draft body is preserved. Bodies exceeding 100,000 bytes are replaced with a link to the tagged changelog during publication; updates are skipped. * Changes to either release notes or changelogs now trigger release-body synchronization. * **Documentation** * Updated release instructions to describe the content selection, size limit, and synchronization behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Marian Koreniuk
|
cd983d416e |
fix(e2e): take the selector's type test in the allowlist scan and pin its regex
lg_suite_has_test dropped every directory with `! -type d`, but the selector keeps a chainsaw-test.y*ml whatever its type, so Chainsaw can fail loudly on a directory by that name. An allowlisted suite in that shape failed validation and stopped the run before Chainsaw reported the real error. Use the selector's own `\( ! -type d -o -name chainsaw-test.yaml -o -name chainsaw-test.yml \)`, which still drops a directory with a step-file name. The step-file regex behind the find glob was never reached by a test: install-01.yaml is already filtered by the glob. 1a-install.yaml passes the glob and not the regex, so it pins the regex; a directory named chainsaw-test.yaml and one named 01-install.yaml pin the type test. The README said the runner recognises the same three layouts. It recognises none: it hands every top-level directory to Chainsaw. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
3d97e912b4 |
fix(e2e): drop the job-level runner.temp and pin the tenant kubeconfig handling
jobs.e2e.env set COZY_PROXMOX_WORKDIR from runner.temp, but the runner context is not available in job-level env, so GitHub would refuse the whole workflow at load time and the Proxmox lane would stop starting. actionlint reports it as `context "runner" is not allowed here`. Drop the variable. run-proxmox.sh already falls back to RUNNER_TEMP, which the runner exports into every step, and the cleanup step removes the file from there. Pin the behaviour the previous commit added, none of which a test covered: the fetched kubeconfig carries no group or other bits under umask 022, apply-tenant removes a file an earlier run left, and the suite's finally removes the file on the keep_tenant path, so diagnostics after it prints no tenant section. Each case goes red with its line removed. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
4d7d336efc |
fix(e2e): tell a failed pool existence read from an absent pool release
The failure-path report checked the pool release with `get hr -o name` and discarded the exit status and stderr, so a timeout, a kill, a Forbidden or a refused connection all printed that the run ended before the worker pool was applied. That is the conflation the guard wrapper exists to refuse, on the path where a slow apiserver is most likely. Read it with --ignore-not-found instead: exit 0 with empty output is an absent release, exit 0 with a name is one to classify, and any non-zero exit is a failed read named through _cozy_guard_read_failure. The pool reads also skipped the report's own deadline and checked only the phase clock, which says yes on every path that opens no phase. Check the deadline before the existence read, like the listing and the addons do, and name what was not read. The stub's ABSENT marker now answers the flag the way kubectl does. New cases pin the failed read and the deadline; the two budget cases are re-timed for the extra clock reading. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
28d9d352e1 |
test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool>
kubernetes-nodes names its release kubernetes-nodes-<CR name> and refuses
to render one that does not start with kubernetes-nodes-<spec.cluster>-
(_helpers.tpl, since
|
||
|
Marian Koreniuk
|
780b068df0 |
fix(kubernetes, proxmox-csi): pin the Proxmox CSI driver images by digest
The CSI controller in the management cluster was referenced by the bare tag ghcr.io/sergelogvinov/proxmox-csi-controller:v0.20.0, and the node plugin in the tenant by the vendored chart's default, its appVersion as a bare tag. That registry is the one whose proxmox-cloud-controller- manager:v0.15.0 was re-pushed to a build that panics, so a tag there is not a statement of what runs. Both now carry the v0.20.0 digest: the controller through images/proxmox-csi-controller.tag, like the cloud-controller-manager, and the node plugin through repository and tag in the proxmox-csi values, which also makes it visible to hack/lib/image-refs.sh and therefore to the release multi-arch gate. Both digests are indexes for linux/amd64 and linux/arm64. Pinning takes the version out of the vendored chart's hands: a chart bump would bring a new appVersion while both pins kept naming the old release, and the controller and the node plugin would drift apart from the templates and from each other without anything noticing. hack/proxmox-csi-driver-pin_test.bats fails unless the chart's appVersion and both pins name the same release. The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI driver: registry.k8s.io publishes them through image promotion, which does not overwrite a tag. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
Aleksei Sviridkin
|
ae9e7c4c64 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on `strategy.name: RetryOnFailure` for both actions, and the e2e HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets the same strategy for the same reason and had no pin. This adds `packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a mirror of `csi_install_retry_test.yaml` rendered on the proxmox substrate, and updates the guard comment that pointed at the gap. No template change. Verified with `helm unittest .` in `packages/apps/kubernetes` (32 suites, 349 tests). Closes #4731 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded automated coverage for Proxmox CSI installation and upgrade retry behavior, including the retry interval and remediation retry limits. The checks verify that retries continue according to the configured settings during installation and upgrades. This update does not change user-facing functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
30fc100e65 |
ci(e2e): end the Proxmox lane's SSH key with a newline (#5090)
## What this PR does The Proxmox lane's Stage step writes `PROXMOX_SSH_KEY` to disk with `printf '%s'`, so the key file ends exactly where the secret value does. `gh secret set` strips the trailing newline from a value it reads from stdin, which is the usual way to set a key (`gh secret set PROXMOX_SSH_KEY < key`). OpenSSH refuses a private key whose last line has no newline: ``` Load key ".../.ssh-pve": error in libcrypto e2e-ro@10.60.0.1: Permission denied (publickey). [20:45:58] FAILED: VM 124 has no scsi0 ``` That is `vm-shape-on-the-hypervisor`, the first test that reaches the hypervisor over SSH. Every run so far stopped before it, so nothing had exercised the key. It showed up on a self-hosted stand once the CCM fix in #4917 let the lane past `tenant-and-cluster`. The step now writes the key with `printf '%s\n'`. If the secret kept its own newline, the file ends in a blank line, and OpenSSH accepts that. ### Testing - `hack/proxmox-teardown-contract.bats` gets a case that runs Stage with a freshly generated key, once without a trailing newline and once with one, and loads the result with `ssh-keygen -y`. It fails against the old step on the first form. The whole file passes: 9/9. - `bats-no-negated-assert`, `bats-strict-setup`, `bats-no-exit-trap`, `bats-runner-coverage`: 63/63. - Reproduced by hand on OpenSSH 9.6p1: the same key loads with one or two trailing newlines and fails with `error in libcrypto` with none. ### Screenshots Not a UI change. ### Downstream repositories The diff changes one workflow step and adds a case to an existing bats file. It moves or renames nothing under `hack/` and changes no make target or `tags.yaml`. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Proxmox workflows now correctly stage SSH keys whether or not the key ends with a newline. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
3fe3028fa6 |
feat(kubernetes-nodes): trust a factory CA in the Proxmox template script (#4860)
## What this PR does
#3797 added `talos.imageFactoryCA`, so a pool's CDI import can verify a
self-hosted Image Factory served by an internal CA.
`hack/proxmox-talos-template.sh` (#4742) downloads the same image with
curl on the Proxmox node. curl there trusts only the system store, so
with that kind of factory it fails with `curl: (60) SSL certificate
problem: unable to get local issuer certificate`, which is exactly the
setup #3797 was written for.
`COZY_TALOS_FACTORY_CA` names a PEM file that holds those certificates,
the same blocks a pool puts in `talos.imageFactoryCA`. The program
writes the node's system bundle into its scratch directory, appends the
certificates after it, and passes the result to `curl --cacert`. They
are therefore trusted on top of the system store, the same way CDI
appends them to its pool, and a redirect to a publicly trusted host
still verifies. Nothing is installed into the node's own trust store.
The certificates are pasted into the program that runs as root on the
hypervisor, so the file is held to the rule
`cozy-lib.tls.validateCACert` applies to the chart value: it must
contain complete certificate blocks and nothing else. The script refuses
a file with a private key alongside the certificates, with trailing
text, or with an unclosed block, and it does so before printing
anything. It also refuses a CA given for a plain-http factory. On the
same principle, curl gets `--proto-redir =https` whenever a CA is set,
so a redirect cannot take the download off verified TLS halfway. Without
the variable, the program is byte-for-byte what it was.
Follows #3797 and #4742.
### Testing
- `bats hack/proxmox-talos-template_test.bats` (Bats 1.14.0) passes
15/15. Three tests are new: the program without a CA is unchanged; with
a CA, curl receives `--proto-redir =https` and the system bundle
followed by the certificates, and CRLF line endings are stripped;
malformed files and a CA for an http factory are refused. The new tests
go red against the script from #4742, against a validator that accepts
any line, and against a bundle that leaves out the system store.
- The same file passes 15/15 under mawk and under busybox awk.
- `bats-strict-setup`, `bats-no-negated-assert` and `bats-no-exit-trap`
are clean, and so is `shellcheck -s sh`.
- The bundle the program writes was checked against a local `openssl
s_server` with a certificate from a throwaway CA: curl verifies with the
bundle and fails with `(60)` without it.
- Against a local HTTPS server that redirects to http, curl follows the
redirect without the flag and stops with `(1) Protocol "http" not
supported or disabled` with it.
- Re-run after the rebase onto main
|
||
|
Myasnikov Daniil
|
b502a9211f |
fix(release): keep release bodies within GitHub's size limit
Finalize sets the release body after the stable tag, the api/apps/v1alpha1 tag and the release-X.Y fast-forward already exist, so a body GitHub refuses fails the step with the release still a draft, and the image retag and chart publish behind it never run. GitHub's web UI cut the v1.7.0-rc.2 changelog at about 125000 characters, and the v1.7.0 changelog is 259,026. A release can now carry a short hand-written docs/releasenotes/<tag>.md that becomes its body, while the changelog stays the full record. Both finalize and update-releasenotes take the release notes when they have content and the changelog otherwise, and hold the body to 100000 bytes. Bytes are never fewer than characters, and the margin makes GitHub's exact limit, and its answer to an oversized body, irrelevant. Over the limit, finalize publishes a one-line link to the changelog on the tag instead, and the sync leaves the published body alone: the tags.yaml backstop brings a changelog to main without its release notes, and the sync would otherwise replace them. A release without release notes whose changelog is under the limit still gets its changelog byte for byte. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
7dc5c0b5e6 |
fix(marketplace): do not install a connected repository's components on connect (#5071)
## What this PR does Connecting a marketplace repository materialized its `PackageSource`s and then auto-created a tap-managed `Package` per source. The Package reconciler installs a `HelmRelease` for every install-marked component of the default variant, and Flux runs those under its `flux` service account, which is bound to `cluster-admin` with no impersonation (`--default-service-account` is not set, and neither `internal/operator` nor `pkg/registry/apps` set `serviceAccountName` on the HelmRelease). A publisher's install chart therefore ran as `cluster-admin` on connect, with no operator step, contradicting the documented behaviour that tapping installs nothing. The `privileged` flag is a prompt to the admin and does not protect against a publisher acting in bad faith. This removes the on-connect auto-registration (`ensureRegistration`). Connecting now only materializes `PackageSource`s. Installing an app is an explicit `cozypkg add`, which becomes the sole creator of a `Package`; every other `Package` reference in the operator is a read or a watch. Removing the call also fixes the second effect of the same path: a `Package` the admin removed with `cozypkg del` is no longer re-created on the next source revision. A second commit reconciles the texts this behaviour change made wrong: they promised that tapping registers a repository's apps in the catalog and runs its charts. Reworded the `cozypkg tap`/`untap` help, the tap success output (it now prints a `cozypkg add <name>` line per materialized PackageSource), the `cozypkg init` scaffold README, the packages-index trust model, and the dashboard Repositories banner and per-repository empty state. Accepted consequence: a connected repository's apps appear in the catalog only after an admin runs `cozypkg add`, since the dashboard catalog is computed from `ApplicationDefinition`s. Install from a connected repository is CLI-only for now; the console connects and disconnects repositories but does not create a `Package`. Populating the catalog from `PackageSource` metadata without an install is the long-term shape in #4359 and is out of scope here. The privileged-confirmation gaps #4358 and #4359 stay open. Upgrade note: a `Package` that Cozystack 1.7.0 auto-created on connect stays after the upgrade, and the privileged-confirmation gate still applies to it, so this is not a regression for existing installs. For a newly connected repository no tap-labelled `Package` is created, so the gate does not come into play on that path. An operator who does not want a 1.7.0 auto-created package can remove it with `cozypkg del`. Addresses #4978. ### Screenshots Dashboard text-only change (Repositories banner and per-repository empty state). No layout change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The diff is operator logic plus CLI/dashboard/doc text; it matches no concrete trigger (no package, API, schema, tooling, annotation, metric or commit-convention change). The dashboard console is vendored in this repo (`packages/system/dashboard`), so its text change travels in this PR. Leaving every box empty rather than ticking "No downstream repository is affected". ### Release note ```release-note fix(marketplace): connecting a marketplace repository no longer installs its components. A connected repository's apps are materialized as PackageSources but not installed until an explicit `cozypkg add`; install from a connected repository is CLI-only for now. Previously, connecting auto-created a Package whose install-marked components were deployed by Flux under its cluster-admin service account. A Package that Cozystack 1.7.0 auto-created on connect stays after upgrade and can be removed with `cozypkg del`. ``` |
||
|
IvanHunters
|
2f39e304f0 |
fix(marketplace): reconcile tap texts with no-install-on-connect
Removing on-connect auto-registration made several user-facing and doc texts wrong: they promised that tapping registers a repository's apps in the catalog and runs its charts. Tapping now installs nothing, so the texts told operators to wait for apps that never appear. Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a `cozypkg add <name>` line per materialized PackageSource), the `cozypkg init` scaffold README, the packages-index trust model, the dashboard Repositories banner and per-repository empty state, and the stale re-registration code comments. No behaviour changes beyond the added `cozypkg add` hint in the tap output. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
749d401899 |
fix(kubernetes, opensearch, foundationdb)!: restrict managed application images (#5085)
## What this PR does
Image overrides let tenants run arbitrary containers in management
cluster, including helpers with access to Talos CA secrets and
Kubernetes API credentials. This PR removes those overrides and uses
images shipped by the platform.
Replaces [the original
PR](https://github.com/cozystack/cozystack/pull/4287) for 1.7. Andrei
Kvapil's commits keep his authorship.
- Kubernetes and KubernetesNodes helpers use the bundled image
references through `cozy-lib.image`. OpenSearch uses its version map.
Talos installer settings for tenant worker VMs stay available.
- FoundationDB uses `version: v7.3` or `v7.4`, mapped to `7.3.63` and
`7.4.1`. Migration 60 carries the old `cluster.version` over to its
release line, records releases needing attention and preserves the
legacy value while the old chart can still render it.
- The FoundationDB chart refuses a render when `cluster.version` and
`version` name different release lines. This also covers variants
without platform migrations.
- OpenSearch and FoundationDB get version-map update scripts. They
retain the current default while it remains supported.
Application types and schemas are still generated from the charts.
### Upgrade notes
Stored image overrides are accepted without new admission warnings and
ignored by the charts. On the first reconcile, OpenSearch nodes move to
the image for their selected version, and Kubernetes components whose
images changed restart. Check custom OpenSearch images before upgrading:
if one runs a newer version than the chart's map, switching back can
leave nodes unable to read their data. Air-gapped installations need
`docker.io` and `ghcr.io` mirrored on management-cluster nodes.
FoundationDB preserves the release line, not an arbitrary patch: 7.3.x
moves to 7.3.63 and 7.4.x to 7.4.1. Check these versions before
upgrading. A cluster on 7.1 needs an upgrade to 7.3 first. On variants
without migrations, set `version` to the line the cluster runs. To
change release lines later, remove the leftover `cluster.version` as
directed by the chart error.
Migration 60 records affected releases in the
`cozystack.io/migration-60-needs-attention` annotation on the
`cozystack-version` ConfigMap. Dry-run reports patch changes without
patching, annotating or advancing the migration stamp. Migration 56 no
longer carries the removed kubectl override into adopted node pools.
An explicit OpenSearch `version: null` now fails rendering instead of
falling back to `v2`.
### Validation
- `make unit-tests` passed, including migrations 56 and 60.
- Four-chart regression run: 750 tests passed.
- API Create/Update, admission, defaulting and existing warning
regression tests passed. `cmd/`, `internal/` and `pkg/` match main.
- Shell test lanes passed with `/bin/sh` as dash throughout the process
tree.
- `make generate manifests` passed with no generated drift, using
cozyvalues-gen v1.7.0.
E2E passed on
|
||
|
Marian Koreniuk
|
70ec326666 |
ci(e2e): end the Proxmox lane's SSH key with a newline
Stage wrote PROXMOX_SSH_KEY with printf '%s', so the key file ended exactly where the secret did. `gh secret set` strips the trailing newline from a value read from stdin, and OpenSSH refuses a private key whose last line has none: ssh reports `Load key ...: error in libcrypto` and the first hypervisor check fails with `Permission denied (publickey)`. Write it with printf '%s\n'. When the secret kept its own newline the file ends in a blank line, which OpenSSH accepts. The contract test now runs Stage with the key both ways and loads the result with ssh-keygen; against the old step it fails on the first. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
myasnikovdaniil
|
432d02e20c |
fix(etcd): stop counting canceled watch streams as failed gRPC requests (#4985)
## What this PR does
The tenant etcd chart has its own copy of
`etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every
`grpc_code` other than `OK` as a failure. `grpc_server_handled_total`
counts a Watch stream once, when it ends, and the streams of a
kube-apiserver that goes away end with `Canceled`. So whenever an API
server on a tenant etcd stops, the Watch method of every member reads as
100 percent failed and the critical rule fires on all of them.
Both rules now count only
`Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`,
the list the etcd mixin has used since etcd-io/etcd#13127 and the one
the platform copy in
`packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The
grouping `BY (job, instance, grpc_service, grpc_method)`, the
thresholds, `for` and the annotations stay as they are, so the alerts
keep their labels.
Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was
created and deleted a few minutes later. vmalert had the critical alert
firing for `grpc_method="Watch"` on all three members with the value
100. At that point
`rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for
Watch with `Canceled`, and the only other non-OK codes were
`FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is
one call each. The same query an hour earlier returned nothing. The
platform copy runs in the same vmalert over the same series, and none of
the firing alerts came from it. Each alert also showed up seven times
there, once per tenant copy of these rules, because the root tenant's
vmalert loads the rules of every tenant namespace. This PR does not
change that.
`hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with
`helm template` and replays it through `vmalert-tool` v1.136.0 pinned by
digest, the same image #4963 uses.
`hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with
seven checks. Watch streams that end only with `Canceled` must not fire
at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25
percent must fire the critical rule at 7 minutes and both rules at 12,
with the labels and the summary pinned. Both together must fire only for
`Range`. Against the rules on main the suite fails 4 of the 7 checks,
all of them on Watch alerts with the value 100. Dropping `Unavailable`
from the list, or grouping by `job` and `instance` only, fails 3 checks
each.
```console
$ bats hack/etcd-grpc-alerts.bats
1..1
ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases
```
The same run on the rules from main prints `not ok 1` and the four
failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes
together with `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`,
`alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like
the migration suites it needs docker, and only the image pull touches
the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run
the new rules in vmalert on a live cluster.
The rules file is the same on release-1.6 and release-1.7. The commit
cherry-picks onto release-1.6 without conflicts, and that backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.
### Screenshots
Not a UI change.
### Downstream repositories
A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and
its test data. No values, schema, README, image or
`ApplicationDefinition` change, and the website does not mention these
rules, so no trigger map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Refined etcd gRPC failure alerts to target specific failure status
codes, reducing alerts for non-failure responses while preserving
existing thresholds and timing.
- Added coverage for canceled watch streams, unavailable range requests,
and combined failure scenarios, including warning and critical alert
evaluations.
- **Tests**
- Added integration validation for rendering etcd alert rules and
running them through the alert-testing tool, including failure handling
when prerequisites are unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Myasnikov Daniil
|
59932e5098 |
fix(platform): report FoundationDB patch drift in migration dry runs
The carry path skipped diagnostics in dry-run mode, so its summary could report no releases needing attention even when the upgrade would lower a FoundationDB patch version. Skip only the write and share the warning and summary bookkeeping with a real run. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
465158f4a0 |
test(platform): load the strict-mode helper in the migration 60 suite
Every unit bats file loads test_helper so its tests run with nounset, and hack/bats-strict-setup.bats fails the unit job on a file that does not. The migration 60 suite passes under it unchanged. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
4cb505a512 |
fix(platform): record 7.1 FoundationDB clusters instead of carrying them
The operator no longer carries the 7.1 client, and the chart no longer offers v7.1, so a 7.1 cluster has no line to be carried to. Migration 59 now records it, together with any release whose version already names a line that is no longer offered; both have to be upgraded or repointed before the chart will render them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
d5ebb62632 |
fix(platform): keep migration 60's record whole across a retry
A failed patch stops the migration before the stamp, and the retry finds version already written on the releases it carried, so it never reached the step that records a patch-level move. The move is now worked out from the values alone, so a retried run records it too. An OpenSearch release storing version: null rendered the old chart's v2 default and is refused by the new one; it now goes on the record before it turns up as a stuck release. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
8c2e20539d |
fix(platform): record every release migration 60 moves
A FoundationDB release carried to its line still moves to the one tag the chart ships for it, which can be a step below what it runs, and that showed up only in the Job log. Kubernetes and KubernetesNodes releases pointing the removed image overrides at a mirror were not listed at all. Both now go into the needs-attention record next to the OpenSearch ones. The migrations image carries no charts, so the tag map is repeated in the migration, and the bats suite fails when it drifts from the chart's versions.yaml. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
033bec74fb |
fix(platform): keep one bad release from stopping migration 60
spec.values on a HelmRelease takes any shape, so a release holding a scalar there, or a cluster.version the apiserver stored as a number, crashed jq and stopped the carry-over for the whole management cluster: every healthy FoundationDB release then went into the upgrade without its version. Those releases are now reported and recorded like any other value that cannot be carried over. The same record covers OpenSearch releases that still set images.opensearch. Nothing can be carried over for them, so what the operator needs is the list. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
f3058d2353 |
fix(platform): guard the FoundationDB version carry-over on all values
Migration 60 tested only cluster.version before adding version, so a version set on the release between the scan and the patch would have been overwritten. The patch now tests the whole values object it read. resourceVersion is not used for this: helm-controller keeps writing the release status, and a mismatch there would abort the platform upgrade for no change to the values. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
930051280b |
fix(platform): carry FoundationDB cluster.version over to version
The foundationdb chart no longer reads cluster.version, so a cluster that set it to 7.1 or 7.4 would move to the v7.3 default on the first reconcile after the upgrade: a full-bounce upgrade from 7.1, or a downgrade from 7.4. Migration 60 sets version to the release line of each cluster.version it finds. It keeps cluster.version, because the old chart still renders the release between the migration and the chart upgrade and would fall back to its own default without it. A value outside the supported lines is recorded on the cozystack-version ConfigMap and does not block the upgrade. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
d62056ef03 |
fix(foundationdb)!: pick the version from a fixed list
cluster.version took any string and passed it to the operator as the FoundationDB version, so a tenant chose the image tag rather than a supported release line. The application now takes version (v7.4, v7.3, v7.1) and maps it to a tag through files/versions.yaml, the same way postgres and mariadb do. The lines are the ones the pinned operator ships client libraries for, which hack/update-versions.sh reads from the operator chart. The map keeps v7.3 on 7.3.63, the previous default, so a cluster that never set the version renders exactly as before. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
873bd6712d |
fix(platform): stop carrying images.kubectl into adopted node pools
Migration 56 copied the parent cluster's kubectl image override into each kubernetes-nodes release it creates. Neither chart reads an images map any more, so the copy only planted an inert key in the pool values. Migration 56 has not shipped in a stable release, so no cluster has run the old mapping outside a pre-release. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Aleksei Sviridkin
|
215e582f03 |
fix(hack): keep the backup stack in the vminstance selective install (#5027)
## What this PR does `vminstance` chainsaw suite has two velero round-trips (`vmdisk-2-backup-roundtrip` and `vminstance-2-backup-roundtrip`), both wait for `cozy-default` BSL that `backupstrategy-controller` package renders together with platform Bucket. `hack/select-install.sh` keeps this package only for database and broker suites ([#4022](https://github.com/cozystack/cozystack/pull/4022)), so PR with selection of exactly `vminstance` gets platform without it: no Bucket, no BucketClaim, no BSL, and both round-trips fail on BSL wait after about 190 seconds, before any step of example scripts runs. Seen on the [E2E run of #4955](https://github.com/cozystack/cozystack/actions/runs/39225333408), selected apps there are `vminstance` only and the backup packages are in disabled list. Any edit of `examples/backups/vmi` or `examples/backups/vmdisk` selects `vminstance` and nothing else, so PRs that change those scripts can't run them. Fix is one more entry in `suite_runtime_sources`, the table that already keeps the stack for database suites. Install closure of `vminstance` grows by `backup-controller`, `backupstrategy-controller`, `bucket-application`, `velero`, `metrics-server` and `monitoring-agents`. Bats test for backup suites now includes `vminstance` and fails without the script change. `hack/select-install_test.bats` and `hack/select-e2e_test.bats` pass (110 tests), `hack/select-install.sh --validate` is clean. Own E2E run of this PR doesn't cover the fix, because `hack/*.sh` escalates to full suite and database suites install the backup stack anyway. Real check is re-run of E2E for [#4955](https://github.com/cozystack/cozystack/pull/4955) after merge, it selects `vminstance` alone. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff, it is one selector table entry and its bats test, no file under `hack/` moved or renamed and no make target changed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(hack): E2E install for `vminstance`-only selection keeps the backup stack, so VMInstance and VMDisk backup round-trips run instead of timing out on BSL wait. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **Bug Fixes** - Virtual machine instances now correctly include the backup strategy component required for backup-related functionality. - Installation checks now consistently validate this requirement, including when backup features are disabled. This improves setup reliability and helps prevent incomplete or misconfigured virtual machine deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d78897f842 |
fix(cluster-api): resync capi-operator every minute (#4939)
## What this PR does After an upgrade restarts capi-operator, a Cluster API provider can keep `Ready=False` for about 20 minutes while its controller is already running, and `capi-providers-core` times out waiting for the CoreProvider. The bundled operator v0.19.0 drops create events of provider Deployments it finds running after a restart (kubernetes-sigs/cluster-api-operator#1031), so it looks at them again only when one changes or on informer resync, and the first resync after a restart comes two sync periods later, about 20 minutes with the default 10m. This sets the operator `--sync-period` to 1m. On a test cluster a provider left `Ready=False` across an operator restart recovered 2m02s after the operator took its lease, the operator with the default period had not recovered after more than 4 minutes. A resync is served from the operator cache and writes nothing while providers are in sync: no provider object was updated over five resyncs, about 15 log lines a minute, 2m CPU and 26Mi memory. The vendored chart has no value for this flag, so a small package patch (reapplied by `make update`) adds one, and a test holds the rendered argument. The upstream fix is in cluster-api-operator v0.27.0, bumping to it is a bigger change and is left for a later release. Mitigates #4926. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(cluster-api): resync capi-operator every minute so a provider left not ready across an operator restart recovers in about two minutes instead of about twenty ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * The Cluster API operator now refreshes its cached state every minute. After an operator restart, provider readiness status can recover sooner instead of remaining stale for a longer period. * Providers that are already in sync are not written back to the API during these cache refreshes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
9a782f697a |
fix(hack): keep the backup stack in the vminstance selective install
The vminstance Chainsaw suite carries the VMInstance and VMDisk Velero round-trips. Both build the cozy-default BackupStorageLocation and the platform bucket from the backupstrategy-controller package, but the selective install retained that package only for the database and broker suites. A selection of vminstance alone therefore installed a platform with no Bucket, no BucketClaim and no BSL, and both round-trips timed out on the BSL wait before running a step of the example scripts. Every pull request whose selection is exactly vminstance hits this, and that includes each edit to examples/backups/vmi and examples/backups/vmdisk, which select that suite on purpose. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
3cd261277e |
fix(backups): keep the backupstrategy-controller Helm release under the Secret size limit (#4941)
## What this PR does Fixes a size regression from #4929. Helm stores each release revision in one Secret and the apiserver refuses a Secret over 1 MiB. With all default Strategy CRs rendered the `backupstrategy-controller` release crosses that limit, so the forced upgrade that creates them fails with `Secret "sh.helm.release.v1.backupstrategy-controller.vN" is invalid: data: Too long: may not be more than 1048576 bytes`, the CRDs stay and the default strategies never appear. The release was already within about 1 KB of the limit before #4929, which added tests to the chart. The strategy CRDs are 2.2 MB of YAML and Helm stores them twice per revision, as chart files and in the rendered manifest. Almost all of it is the upstream `PodTemplateSpec` schema inlined by the Altinity, Job and FoundationDB strategies. Kafka, RabbitMQ and Redis already mark their `Template corev1.PodTemplateSpec` as `Schemaless` and `PreserveUnknownFields`, this PR does the same for `AltinitySpec.Template`, `JobSpec.Template` and `FoundationDBTemplate.BackupDeploymentPodTemplateSpec`. Go types are unchanged and every description is kept. The CRDs go from 2.2 MB to 317 KB and the largest release revision from over the limit to about 250 KB, 24% of it. The size rationale now lives once in the package doc, the field docs of the six schemaless templates share one short line. What changes for users: the apiserver no longer validates these three fields and `kubectl explain` no longer lists children under them (`kubectl explain pod.spec` still documents them), same as for Kafka, RabbitMQ and Redis today. Unknown keys are stored instead of pruned, the typed decode in the drivers drops them either way. Labels and annotations under `template.metadata` were rejected by the strict v1.6 schema and are now accepted and kept. A Go template in a non-string slot never worked, it was rejected on apply, now the apiserver accepts it and the typed decode in the controller is what rejects it, as for the other three kinds (I did not run that case live). The drivers read these fields through the non-strict client and `internal/template` rewrites only string leaves, nothing under `internal/backupcontroller` relies on pruning. Go tests for `internal/backupcontroller`, `internal/template` and `api` pass. `hack/strategy-crd-schemaless.bats` fails on any strategy CRD that inlines a pod schema and prints the property path and the two markers to add. `hack/helm-release-size.bats` estimates the stored release of every chart under `packages/` the way Helm encodes it (release JSON, gzip, base64) and fails above 90% of the limit, plus the worst case of this chart. The estimate runs 1 to 2 percent above real Secrets. A chart has to render offline to be measured with its manifest: the 36 charts that do not are listed in `NO_OFFLINE_RENDER` and measured on files and templates only, any other render failure fails the test, and so does a listed chart that starts rendering. Checked on a throwaway kind cluster: v1.6.4 upgrades to this branch and the forced follow-up creates all default strategies, a fresh install works, strategy objects created under the old schema stay valid, a Job strategy with templated args and pod template labels is accepted. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(backups): the backupstrategy-controller release no longer fails with "Secret ... Too long" on the upgrade that creates the default backup strategies, the Altinity, Job and FoundationDB strategies store their pod templates schemaless like Kafka, RabbitMQ and Redis ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Backup strategy templates for Altinity, FoundationDB, Jobs, Kafka, RabbitMQ, and Redis now preserve fields not defined in the API schema. The API server no longer validates these templates against the pod schema; validation occurs when the driver creates the workload. This lets strategies include pod-template fields without requiring those fields to be listed in the strategy schema. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
c6fcb4e7c6 |
fix(ci): dispatch frozen-line builds with GITHUB_TOKEN (#4888)
## What this PR does The first RC freeze hit HTTP 403 when dispatching `build-release.yaml` because the CI app has no Actions write permission ([run](https://github.com/cozystack/cozystack/actions/runs/38229949424)). I switched this dispatch to `GITHUB_TOKEN` with job-scoped `actions: write`. Tag and branch pushes still use the app token. Verified with `make unit-tests`, zizmor 1.26.1 and the commit trailer check. The regression test fails on the old workflow and passes with the fix, including warning and manual recovery on dispatch failure. Live dispatch with `GITHUB_TOKEN` has not been tested. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ci): use GITHUB_TOKEN to start the initial packages build when freezing a release line ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated release automation to use appropriate permissions and authentication for CI triggers. * Improved warnings and recovery guidance when automated package builds cannot be started. * **Tests** * Added automated checks covering release dispatch permissions, branch targeting, successful execution, and non-blocking failure handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
a6cb84673e |
docs(release): check changelog claims against the code (#4936)
## What this PR does Changelog generator sees only `docs/agents/changelog.md`, two inline prompts and `AGENTS.md`. Failure modes and pre-publish checks live in `docs/release.md`, which it never reads. The prompt has detailed rules for authors and PR numbers and none for checking a claim against the code. I reviewed the generated v1.7.0 draft against the code and found 22 statements wrong or missing, while all authors and PR references were right. Several were in breaking changes: a talm fix credited to a talm release that doesn't contain it, worker pools said to be adopted without churn while Talos workers are rolled once because of the hash-named TalosConfigTemplate (#3523), migrations said to run on a variant that has them disabled. Three changes: - `docs/agents/changelog.md`: new section 8b about verifying claims against the code. Identifiers, migration behaviour and variant scope come from the tree at the upper-bound tag, required actions come from release notes and newly added upgrade sections, not only from `!` commits. Section 9 now asks to run the `docs/release.md` checks and `hack/validate-changelog.sh` on the written file before exit. - `.github/workflows/changelog-generate.yaml`: refresh `docs/release.md` and `hack/validate-changelog.sh` from the dispatch ref, next to the existing `changelog.md` refresh. Job checks out the rc tag detached, so without this generator runs older copies from that tag. Permissions, tokens and tool flags are unchanged. - `hack/validate-changelog.sh`: reject an entry cut off with an ellipsis right before its attribution, and any link to `cozystack.io/docs/next/` (live site answers 404). Reason names the offending lines in one line, because workflows quote it into a single annotation. How I checked it: - Old validator accepts the original v1.7.0 draft, new one rejects it on three truncated entries (lines 335, 393, 518) and, after those are repaired, on the `/docs/next/` link (line 25). All 29 shipped changelogs from v1.3.0 to v1.6.4 still pass. - New contract test plants each defect into a shipped changelog, checks the reason names the line, and checks that near misses real changelogs carry still pass (`content/en/docs/next/` in prose, ellipsis inside a sentence, three dots of a compare link). It goes red on the old validator and on widened patterns. - Overlay test pins both new copies and the cone-mode sparse checkout that brings `docs/release.md` in. It goes red if either copy is removed, cone mode is turned off or `docs/agents` is dropped from sparse paths. - I ran every command the new prompt rules prescribe on v1.6.0..v1.7.0-rc.1, each one finds an error the draft shipped. - `make bats-unit-tests bats-posix-compat-tests` pass (2424 and 1120), zizmor and actionlint are clean on the workflow. Not included: a script that gives the generator component versions and migration headers, and a networked check for references, authors, link liveness and companion-release containment. Both are separate changes. Things I noticed and left alone: - validator summary in the Phase 2 section of `docs/release.md` doesn't list the two new rejections - `docs/agents/changelog.md` runs `docs/changelogs/hack/check-optional-repos.sh`, but the script lives at `hack/check-optional-repos.sh` and still resolves repo root from the old location - a changelog the validator rejects is still not uploaded as an artifact, so rejection costs a regeneration or a hand-commit, that is why section 9 has the generator run the validator first - current v1.7.0 draft still has the three truncated entries, a promotion that picks it up will discard it and generate a new one unless the entries are rewritten - three unrelated bats tests (in `check-gpu-recording-rules.bats` and `promote-rewrite-tags_test.bats`) fail under en_US.UTF-8 collation on main too, they pass under C.UTF-8 that runners use, I ran the suite that way ### Screenshots Not applicable, there is no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I walked the trigger map in `docs/agents/contributing.md` against each changed file: - `docs/agents/changelog.md` is the changelog prompt, not the commit convention or the PR template that ccp and community restate. Its list of release-window repositories is unchanged. - `.github/workflows/changelog-generate.yaml` is not `tags.yaml`, the release-prep workflow ccp relies on. - `hack/validate-changelog.sh` is changed in place, not moved or renamed, and it is not `hack/package.mk` or `hack/update-crd.sh`. - `hack/release-changelog-contract.bats` is a test. No package, values key, CRD, release asset, make target or node prerequisite changes. Code search across the organization finds no reference to any of these files outside this repository. ### Release note ```release-note docs(release): check changelog claims against the code. The changelog generator now verifies operator-facing statements against the code and runs the release checks on its own output, and the changelog validator rejects entries cut off with an ellipsis and links to cozystack.io/docs/next/. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Release guidance now emphasizes verifying operator-facing claims against the release version and carrying required actions into upgrade instructions. * Documentation links in changelogs should point to versioned documentation. * **Chores** * Changelog validation now flags truncated entries and links to unversioned “next” documentation. * Release guidance and validation checks are refreshed automatically during changelog generation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
f69958b15d |
docs: move the operator guides out of docs/ to where readers find them (#3780)
## What this PR does
Five documents under `docs/` were operator facing guides written
alongside the feature PRs that added the code, and left in the source
tree where no user looks. They are on the website now, so this deletes
them here and repoints every reference.
```
docs/oidc-tenant.md -> kubernetes/oidc-authentication
docs/oidc-grafana.md -> operations/services/monitoring/oidc-authentication
docs/gpu-vgpu.md -> virtualization/vgpu
+ operations/troubleshooting/gpu-operator-host-driver
docs/vm-external-vlan.md -> networking/vm-external-vlan
docs/hubble-observability.md -> networking/hubble
```
`docs/vm-external-vlan.md` needed splitting, not moving. Its "Staging
the reference CNI plugins" section is a contract,
`hack/cni-plugins-staging-contract.bats` greps six literal sentences out
of it and asserts them against the multus Dockerfile, the daemonset
manifest and the platform values. That half is package documentation so
it moves verbatim into a new `packages/system/multus/README.md` and the
suite reads it there, the operator guide goes to the website. The bats
messages name the README now, so a future failure points at the file
that has to change.
References repointed in the `@param` blocks, template comments and test
comments that named the guides. The generated files come from `make
generate` with cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`. Where a reference points at text
that exists only on `next/` of the website (the aggregated API servers
section of the tenant OIDC guide), the comment names the page and the
section instead of a `v1.6` URL, which would not contain it.
Two changelog links in `docs/changelogs/v1.6.0.md` repointed as well.
The wording is untouched, only the location of the documents it names
has changed.
One bit of process framing dropped on the way past, per #3777:
`bundles/system.yaml` said GrafanaAdmin promotion was "out of scope for
Phase 1" when it is simply not implemented.
Tests: 10 cni-plugins contract, 47 install-wait, 346 kubernetes chart,
22 multus chart, 301 monitoring chart, rd-presets-check.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#648 (the four guides),
cozystack/website#754 (edits made to the originals in `docs/` after the
import), cozystack/website#647 (grafana guide made reachable in v1.6),
cozystack/website#596 (tenant kubectl guide, rebased)
Walked the rest of the trigger map against the diff. No provider
follow-up: the kubernetes `values.schema.json` diff is two `description`
strings and the monitoring one is one, with no field added, removed or
renamed, no enum, no default, and no `kind`, `plural` or
`release.prefix` change. `packages/core/platform/values.yaml` changed in
a comment only and no key, so the hand written platform-package table on
the site is unaffected.
Please merge after cozystack/website#754 and cozystack/website#647. The
`@param` URLs only resolve once those pages deploy, and
cozystack/website#754 carries the edits made to the guides since the
import, which deleting them here would lose.
### Release note
```release-note
docs: operator guides for tenant Kubernetes OIDC, Grafana OIDC, NVIDIA vGPU, VM external VLAN and Hubble now live on cozystack.io instead of the docs/ directory of this repository, and in-repo references point there. The CNI plugin staging contract that shared the VLAN guide moved to packages/system/multus/README.md.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Updated configuration and test references to link to the published
Cozystack documentation.
* Added Multus guidance covering staged CNI plugins, configuration,
troubleshooting, and rollout behavior.
* Removed outdated local guides for GPU/vGPU, Hubble observability, OIDC
authentication, and external VM VLANs.
* Clarified that server-level Grafana administrator promotion is not
currently supported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
1f92bcd20e |
fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings (#4914)
## What this PR does Follow-up to the flux-plunger uninstall fence: the points the last review round raised and left for after the merge, plus four found by a later independent review. None changes the fence's design; each closes a gap in what it reports, how long it waits, or what its documentation says. **Pre-delete hook.** - A release being deleted that only the fence still holds is no longer unfenced in the same step that lifts its suspension. helm-controller runs the uninstall of a release being deleted once it is no longer suspended, but removing the fence at once deleted the object before helm-controller could reconcile it. The hook now keeps that fence while helm-controller handles the generation the lifted suspension wrote, until shortly before its retry window closes, and names a release it still had to let go of as before. A release whose state cannot be read keeps its fence for the wait the same way, rather than being let go of on one failed read. - A release being deleted while still suspended, by anyone but flux-plunger, was reported as cleanly released, although helm-controller skips the uninstall of a suspended release. It is now named as deleted without an uninstall and counted with the others. - Each release gets its own share of the retry window, capped at the window's end. One release that could never be read used to take the whole window and leave every release listed after it a single attempt. The share is a cap, not a floor of attempts: a floor would let enough failing releases run the Job past its deadline, which kills it before its report is written. - The hook stops, waits for and lists again only for the flux-plunger of its own release. Another one (a second installation under another release name, changed labels, a binary outside the cluster) is not stopped by the hook, keeps the fences it sets and gives them up itself; the README now says so instead of the hook re-listing unconditionally, which would not close that case and would fail the uninstall for as long as a second installation is recovering anything. - helm-controller's finalizer, which the hook reads to tell whether a release being deleted is still held, is tied to the constant the controller uses, as the fence already was. Two comments are corrected: the retry window counts from the script's start, not the Job's, and kubectl scale's internal conflict retry never runs here. **Shutdown drain.** - A drain deadline of zero or less is refused at startup, and the chart's lowest accepted grace period is held to the same constants as the drain subtraction, so raising the manager's stop bound cannot make the chart render a deadline that is already spent. The timings the README and the values comment state are held to those constants too. - Fences left behind are reported as an error naming each release, not an Info line; a drain that never got a listing answered says so instead of reporting a count of -1; a flux-plunger that never led no longer claims it released every fence. - The drain's Events fold repeats into one counted Event per release and reason, as the manager's recorder does, instead of one object per pass. The fold is a patch, the verb the chart grants on events, and every write is bounded by the drain's deadline. It does not add a per-release Event for fences left at the end: by then the drain's deadline is spent, and writing one would reach past it. - Tests now hold the outcome line, the retry of a release whose fence cannot be given up yet, and the pause between passes, none of which any suite noticed when removed. **Controller.** The `SuspensionNotOwned` and `TerminatingWhileSuspended` Warnings were keyed on the processed-version annotation, which stays on every release flux-plunger ever recovered, so a later restore, migration or operator suspension of a healthy release drew a Warning telling the reader to resume it, and a migration's deliberate suspend-then-delete was called an accident. Both now key on the state a leftover actually freezes the release in, `Ready=False` with `has no deployed releases`, since helm-controller rewrites no condition of a suspended release; the README's candidate query follows. Two comments that claimed more than holds are qualified. **Chart and README.** A `shutdownGracePeriodSeconds` of a million or more now gets the range message rather than being refused as a fraction. The README gives the recovery steps in the order the controller runs them (the annotation is written before the Secret is deleted), and names the window in which a backup restore started during a recovery is unsuspended under it, since the etcd, CNPG and Velero restore steps write `spec.suspend` only when it differs. ### Testing Each change was written test first: a test that builds the binary and runs it with a deadline of zero, helm-unittest for the chart, unit tests with a captured logger and the fake client for the drain's outcome, Events and passes (the Event fold under a client that refuses the update verb, as the chart's role does), table tests for both Warnings, and the bats suite for the hook, whose fake kubectl now models helm-controller handling a release some reads after its suspension is lifted and a state read that fails. Every fix was then reverted on its own, and the suite went red each time. The envtest suites pass unchanged. The README's candidate query was run against HelmReleases read from a live cluster, with a stuck, a healthy, an owned and an unsuspended variant, and lists only the stuck one. ### Screenshots Not applicable: no UI change. ### Downstream repositories The trigger map was walked against the diff: `internal/controller/fluxplunger`, `cmd/flux-plunger`, the `packages/system/flux-plunger` chart (templates, tests, README) and its bats test under `hack/`. The website's FluxPlunger section in `guides/concepts.md` draws the recovery with the Secret deleted before the annotation is written, which is the order before the fence change; it needs the same correction as the chart README. That change is held as a draft until v1.7.0 is released and the website's `next/` is re-initialized: the fence change is not part of release-1.7, so the corrected order would be wrong for v1.7 docs, which are copied from `next/`. No other repository is reached by this diff. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/752 (draft, merge after v1.7.0) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(fluxcd): flux-plunger's pre-delete hook now lets helm-controller run the uninstall of a release being deleted before it removes the release's fence, gives each release its own share of its retries, and reports releases deleted under another manager's suspension; the shutdown drain reports the fences it leaves as errors and refuses a deadline that leaves no time ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Shutdown drain timeouts must be positive, and invalid values now stop startup with a clear error. * Uninstall cleanup waits for Helm to handle releases before removing their fences, while bounding retries and reporting releases that remain incomplete. * Recovery and suspension warnings now reflect the release’s actual failure state, and recovery can retry stale storage cleanup against the same revision. * **Documentation** * Clarified shutdown grace-period requirements, uninstall behavior, and guidance for backup restores and existing suspensions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
d1d4e92acb |
feat(release): sign the release assets and attach a source SBOM (#4861)
## What this PR does Implements the release-asset part of #4759: a release cut from a tree that has this change gets an SBOM of its source and one keyless signature that covers all of its assets. These are the two CLOMonitor checks still failing for this repository, `sbom` and `signed_releases`, after #4757 and #4899 brought the project score to 96.69. `hack/upload-assets.sh` now collects its asset list into an array and hands it to the new `hack/sign-release-assets.sh` before the first upload. The script writes three files into `_out/assets`, and they are uploaded with the rest: | File | What it is | |---|---| | `cozystack-source-sbom.spdx.json` | SPDX 2.3 SBOM made with syft, over `git archive HEAD` (the committed release tree, without `_out/`, `.release-tooling` or runner leftovers) and the unpacked `cozypkg-*.tar.gz` binaries. It does **not** describe the container images that `packages/` pins by digest (their SBOMs are #2937, A2 in #3278) or the Talos ISO, disk, kernel and initramfs images. It is named by what it covers, so `cozystack-sbom` stays free for an SBOM of the release as a whole. CLOMonitor's `sbom` check matches `sbom` in an asset name. | | `cozystack-checksums.txt` | SHA-256 of every uploaded asset and of the SBOM. | | `cozystack-checksums.txt.sigstore.json` | Sigstore bundle of `cosign sign-blob` over the checksums, keyless, signed with the job's GitHub OIDC identity. | Before it returns, the script runs the same `cosign verify-blob` that `docs/release.md` gives users, with `--certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)'`: `tags.yaml` on a `v*` tag push, `promote-rc.yaml` dispatched from `main`. A signature that check would reject therefore fails the job before anything is uploaded. It also refuses its own three outputs as inputs: `promote-rc.yaml` downloads every rc asset, and a stable release has to get freshly computed checksums and a fresh signature, since promotion restamps several assets. Workflows: `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, the two jobs that call `make upload_assets`, get `id-token: write` and two setup steps: - syft v1.54.1 from its release tarball, checked against a committed SHA-256 (`c069905b…`, the release's checksums entry; the checksums file verifies against anchore/syft's keyless release signature). Not `anchore/sbom-action/download-syft`: pinned by commit, it still runs syft's `install.sh` fetched from a tag at run time, in a job that holds the app token, the GHCR login and `id-token: write`. v1.54.1 is past GHSA-mw2c-m758-9v5q (fixed in v1.52.0). `hack/ci-tool-installs-contract.bats` now fails on `anchore/sbom-action`, on a syft install script, on a syft download without `sha256sum -c` and on version or checksum drift between the two workflows. - `sigstore/cosign-installer` v4.1.2, pinned by SHA, with `cosign-release: v3.1.3`: the version `docs/release.md` asks users for, so the job checks its own signature the way they will. The action verifies that binary against cosign's keyless and KMS bundles with its digest-pinned bootstrap cosign. `hack/ci-tool-installs-contract.bats` holds the action commit and the release equal in both workflows and fails on a step without `cosign-release`. `docs/release.md` gets a "Release signatures and SBOM" section with the user-side verification: cosign v3.1.3 or later (v2.6.5 on the v2 line), the two commands joined with `&&`, what the SBOM covers and what the certificate identity proves. It now also says, at each place `promote-rc.yaml` is run, that it is dispatched from `main` only. ### Decisions for the release owners - **Fail closed.** If Fulcio or Rekor is down, the release job fails and nothing is uploaded. I chose that so that no release goes out unsigned. If you would rather publish unsigned than block, the change is a `|| echo ::warning::` around the call in `upload-assets.sh`. - **Promotion runs the rc tree's script.** `promote-rc.yaml` calls `make upload_assets` from the rc checkout, not from `.release-tooling`. The first signed stable release is therefore the promotion of the first rc tagged after this merges. `release-1.6` gets signatures only if this is backported. CLOMonitor reads the latest release, so the score moves only when that release carries the files. - **Certificate identity pins the refs** (decided in review): `tags.yaml@refs/tags/v…` and `promote-rc.yaml@refs/heads/main`. A `promote-rc.yaml` dispatched from another branch fails the job's own check at the upload step, after the staging branch, the `promotion-*` candidate and the empty draft release exist; nothing in `parse` refuses a non-`main` dispatch up front. Until the `v*` tag rulesets exist, a hand-pushed `v*` tag whose tree has an edited `tags.yaml` still signs, which `docs/release.md` says. - **Follow-ups after the first signed release** (#4759 stays open for them): - `project.documentation.signature-verification` in `security-insights.yml` (#4757); - the SBOM row in `docs/security/self-assessment.md`, which says "Not published at this time"; the image caveat above stays in it. I left both alone so that they do not describe releases that don't exist yet. ### How it was checked - Bats 1.14.0 (the version CI installs), with gh, syft and cosign stubbed: - `hack/sign-release-assets.bats`, 8 tests: what the checksums cover; that syft sees the committed tree and the cozypkg binaries but not untracked files; that the verify identity accepts `tags.yaml` on a `v*` tag and `promote-rc.yaml` on `main`, and rejects a fork, another workflow, `tags.yaml.evil`, `tags.yaml` on a branch, and `promote-rc.yaml` on `release-1.6`, on `main-evil` and on a tag; that `docs/release.md` shows the same regexp as the script; that a failed verification fails the script; that it refuses a missing asset, assets split over two directories (with a same-named file in both, so the test fails if the directory check is removed) and its own outputs as input. - `hack/upload-assets.bats`, 4 tests, new: the exact upload list and order (the assets, then the SBOM, the checksums and the bundle); that the checksums name every upload but the last two; zero `gh` calls when verification fails or an asset is missing. The old `upload-assets.sh` fails all four, a version that signs after uploading fails the last two. - `hack/ci-tool-installs-contract.bats`, 8 tests. Putting `download-syft` back, dropping the `sha256sum -c` line, letting `SYFT_VERSION` drift between the workflows, dropping `cosign-release` from one step, letting it drift, unpinning the installer or giving one step another installer commit each fails it. - Putting the old any-ref regexp back in the script fails the identity test and the docs tie; putting it back in the docs alone fails the tie. - The meta-suites `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage` and `bats-runner-contract` pass: 90 tests across these eight files. - The pinned regexp gives the same accept/reject results under `grep -E` and Go's `regexp` (what cosign uses). cosign-installer's install script at the pinned commit, run as written with `cosign-release: v3.1.3`, verifies the binary's keyless and KMS bundles and installs v3.1.3 (its SHA-256 matches `cosign_checksums.txt`). `--yes`, `--bundle`, `--certificate-identity-regexp` and `--certificate-oidc-issuer` are listed, not deprecated, in cosign v3.1.3's help. - The `Set up syft` step's script, run as written with `RUNNER_TEMP`/`GITHUB_PATH` set: the checksum passes and `syft version` reports 1.54.1. A real syft v1.54.1 run through `hack/sign-release-assets.sh` (cosign stubbed) over this tree and the v1.6.5 `cozypkg-linux-amd64.tar.gz` took 10 s and wrote an SPDX 2.3 document with 920 packages: 440 golang, 373 npm, 103 github, 4 without a purl; 60 come from the cozypkg binary. - cosign behaviour behind the docs, with a key-signed legacy bundle whose `cert` holds the bare public key, against the documented identity regexp (tlog ignored, since an attacker can upload their own entry): v2.6.4 and v3.1.2 print "Verified OK", v2.6.5 and v3.1.3 reject it. `--new-bundle-format` makes v2.6.4 reject it but not v3.1.2, so the docs give the minimum version and not the flag. A real keyless protobuf bundle (syft's) verifies on v2.6.5 and v3.1.3. - zizmor 1.26.1 `--offline` reports no findings. actionlint 1.7.12 (with `.github/actionlint.yaml`) reports the same 13 pre-existing info-level shellcheck notes in `tags.yaml`/`promote-rc.yaml` as before, nothing new. shellcheck passes on both scripts. `hack/check-commit-trailers.sh` passes. - Not checked: a real keyless signature. It needs the OIDC token of a job in this repository, so the first rc after the merge is the end-to-end test. Cutting an `-alpha.N` first would exercise it without touching an rc. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: I checked the trigger map file by file. The only entry `hack/upload-assets.sh` can trigger is renaming a release asset, because the website install pages hardcode asset names. This PR adds three assets and renames none of the released ones, `cozypkg-checksums.txt` included (the SBOM was renamed to `cozystack-source-sbom.spdx.json` in review, before any release carried it). The workflow and docs changes match no entry. ### Release note ```release-note feat(release): releases carry cozystack-source-sbom.spdx.json, an SPDX SBOM of the source tree and the cozypkg binaries (not of the container or Talos images), cozystack-checksums.txt with the SHA-256 of each asset, and cozystack-checksums.txt.sigstore.json, a keyless Sigstore signature over the checksums made by the release workflow; docs/release.md shows how to verify a download with cosign v3.1.3 or later. ``` Refs #4759 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Releases now include an SPDX software bill of materials, SHA-256 checksums, and a keyless Sigstore signature bundle. * Release assets are verified against the expected workflow identity before upload; uploads stop if verification fails. * **Documentation** * Added guidance on release signatures, SBOM coverage, asset verification, and which branch dispatches release workflows. * **Tests** * Added coverage for generated release files, upload behavior, and signing and verification failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Yan Bondarenko
|
1ca7c0d891 |
fix(etcd): stop counting canceled watch streams as failed gRPC requests
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted every code other than OK. A Watch stream is counted once, when it ends, and the streams of a kube-apiserver that stops end with Canceled. So whenever an API server on that etcd goes away, the Watch method of every member reads as 100 percent failed and the critical rule fires on all of them. The rules now count the codes the etcd mixin counts since etcd-io/etcd#13127, the same list the platform copy in monitoring-agents already uses. The grouping by method stays, so the alert labels do not change. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
Marian Koreniuk
|
782e0ab64d |
fix(e2e): validate the leak allowlist against every suite layout
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the selectors already see step-file suites and a Test nested below the suite directory. A suite in one of those layouts could not be allowlisted at all: the run stopped before the first suite. Validate the allowlist with the rule hack/select-e2e.sh enumerates suites with, spelled the same way on both sides and cross-referenced so they move together. Describe the three layouts once in the e2e-chainsaw README, point the agent doc at it, and list kubernetes-proxmox as the parked suite. Closes #4659 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
61238500dd |
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it
run-proxmox.sh wrote the tenant's super-admin kubeconfig to $TMPDIR with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. dump_diagnostics then read whatever sat at that path, so a run that failed before the fetch reported the previous run's tenant as its own. The file now lives under COZY_PROXMOX_WORKDIR, which the workflow points at runner.temp. It is written under umask 077, removed at the start of apply-tenant so this run never sees a leftover, removed again in the suite's finally, and once more in the workflow's credential cleanup step for a run killed before finally runs. Closes #4806 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
cac152048e |
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard
cozy_guard_all_helmreleases reads the parent and the kubernetes-<test>- addons. The worker pool is kubernetes-nodes-<test>-md0, an Application HelmRelease the apiserver stamps with RetryOnFailure like the parent, on the node-join path through the talos-reconcile Job it renders. Its name does not start with the addon prefix, so a failed action on it was never read, on the passing path or in the failure-path report. Read it on both paths. The pool name is derived from the parent, the way the addon prefix is, so the two cannot drift. The failure-path report checks that the release exists first: a run that failed between the parent and the pool apply has none, and that is a fact about where the run stopped rather than a failed read. Closes #4737 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
98d0402096 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml sets the same strategy for the same reason and had no pin. Add the mirror test and update the guard comment that pointed at the gap. No template change. Closes #4731 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
bc24b588b2 |
feat(release): sign the release assets and attach a source SBOM
No release carried a signature or an SBOM, so users could verify none of the ISO, disk images, kernel, initramfs, manifests and cozypkg binaries they download, and the OpenSSF Scorecard Signed-Releases check and CLOMonitor's signed_releases and sbom checks failed. hack/upload-assets.sh now hands its asset list to the new hack/sign-release-assets.sh before the first upload. That script writes cozystack-source-sbom.spdx.json with syft over the committed tree and the cozypkg binaries, cozystack-checksums.txt with the SHA-256 of every asset and the SBOM, and cozystack-checksums.txt.sigstore.json with a keyless cosign signature over the checksums. The SBOM does not describe the container images or the Talos boot images, which is what its name says; image SBOMs are #2937. The script verifies the bundle against the identities of tags.yaml on a v* tag and promote-rc.yaml on main before returning, so a bad signature fails the job with nothing uploaded, and hack/upload-assets.bats pins the upload list and that order. The two jobs that call make upload_assets get id-token: write, cosign v3.1.3 from the SHA-pinned cosign-installer, and syft from its release tarball checked against a committed SHA-256, because anchore/sbom-action/download-syft runs an install script fetched at run time; hack/ci-tool-installs-contract.bats now covers both. docs/release.md describes the files, what the SBOM covers, the minimum cosign version to verify a download with, what the certificate identity proves, and that promote-rc.yaml is dispatched from main only. Refs #4759 Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
Myasnikov Daniil
|
7ad7d39349 |
test(hack): keep every chart's stored Helm release within budget
A chart can grow past the 1 MiB Helm release Secret without any test noticing: it renders, lints and installs, and only the upgrade whose manifest is large enough fails, on a live cluster, with a release that stays failed. backupstrategy-controller sat under 1 KB from the limit before it crossed it. Estimate the stored size the way Helm encodes it (release JSON with the chart's templates and files, gzip, base64) and fail any chart under packages/ above 90 percent of the limit, plus the largest revision of backupstrategy-controller with every default Strategy rendered. The estimate counts files a .helmignore names, because the production path stores them. A size measured without its manifest is a lower bound that passes for the wrong reason, so every chart must render offline except an explicit list of charts that need cluster state, and the worst-case render must succeed and contain the default strategies before it is measured. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
513b5088a8 |
test(backups): fail on a strategy CRD that inlines a pod schema
The release-size budget only reports that the backupstrategy-controller release grew too large, not which field did it, and three strategies once inlined the upstream pod schema before anyone noticed. Check the generated strategy CRDs for an inlined PodSpec and name the property path, with the markers that fix it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
c2dab12a2e |
fix(cluster-api): resync capi-operator every minute
A restarted capi-operator v0.19.0 drops the create events for the provider Deployments it finds already running (kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready condition went False while the operator was down therefore keeps it until that Deployment changes or the informer resyncs, and the first resync after a restart lands two periods later because the handler registers after the informer has listed. With the 10m default that is about 20 minutes, during which capi-providers-core times out on the CoreProvider and everything depending on it waits; every further operator restart starts the clock again. With a one-minute period the same repro recovered 2m02s after the operator took its lease, against no recovery within 4m26s with the default on the same cluster. A resync replays the cache and, with providers in sync, ends in "No changes detected" and an unchanged condition, so it writes nothing to the API server; the operator logged about 15 lines a minute and used 2m CPU and 26Mi of memory. The upstream chart has no value for the flag, so a package patch that make update reapplies adds one, and a bats test holds the rendered argument. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |