mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 05:25:14 +09:00
f035da8a0f9eeeb2d9c2de74ebd3972ab7cc485c
5715
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 IvanHunters
|
f035da8a0f |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Resolve conflicts introduced by main: - Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in favour of the GPU type main added; main removed the fluxcd addon wholesale. - Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57. - Accept the branch's removal of the in-chart talos-reconcile Job; it moved to the kubernetes-nodes chart, main only touched its comment. - Regenerate kubernetes values.schema.json / README and the kubernetes-rd ApplicationDefinition schema. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
adc9c568fd |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs (#3887)
## What this PR does Break the tenant `bootstrap-token-tenant` Job out of its `helm.sh/hook: post-install,post-upgrade` lifecycle and move it into the main install phase, so the `bootstrap-token-<id>` Secret lands in the tenant `kube-system` before `MachineDeployment` can converge. ### Why it deadlocks today - Job wired as `post-install,post-upgrade` fires only after Helm's wait phase resolves - Wait phase blocks on `MachineDeployment` Ready - MD Ready blocks on nodes registering - Node registration blocks on the kubelet CSR path, which needs the very `bootstrap-token-<id>` Secret this Job produces - Result: any fresh install, or any HR reconcile that finds MD in a non-Ready state, sits in a deadlock; `HR.status.upgradeFailures` climbs without the post-upgrade hook ever getting its turn `pre-install,pre-upgrade` is not a fix either — those fire before non-hook templates apply, so `<release>-admin-kubeconfig` (materialised by Kamaji after `KamajiControlPlane` is applied) is not yet available. ### The shape that works Same shape `talos-reconcile` already uses in this chart: main-phase resources. Helm applies the SA and Job together with `KamajiControlPlane` and `talos-secrets`. Job's Pod stays pending until Kamaji publishes the admin kubeconfig Secret (`backoffLimit` covers the retry window). Job completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state — it doesn't gate it on MD. ### Verification - Reproduced the deadlock on two independent long-running tenant clusters (v1.6.x), documented in #3875 - Reproduced the manual unblock (creating the `bootstrap-token-<id>` Secret by hand in the tenant `kube-system`) on both — nodes register within seconds, MD converges within minutes, HR clears - Extended `packages/apps/kubernetes/tests/talos_templates_test.yaml` to pin the absence of `helm.sh/hook` on both rendered documents; `helm unittest` passes 9/9 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md`. The diff touches `packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml` and its test — no app add/remove, no `values.schema.json`, no core/platform values, no `hack/`, no CRD, no `ApplicationDefinition`, no telemetry metric. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): tenant `bootstrap-token-<id>` Secret is now created in the main install phase instead of a post-install/post-upgrade hook, unblocking fresh installs and any HR reconcile that finds MachineDeployment in a non-Ready state. ``` Closes #3875 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved bootstrap-token job scheduling by running it as a standard chart resource instead of a Helm lifecycle hook. * Jobs are automatically recreated when relevant configuration changes, ensuring updates are applied reliably. * Added startup handling to wait for the admin kubeconfig when needed. * Improved retry and cleanup behavior so deployments are not unnecessarily blocked. * Ensured related resources are created consistently during chart deployment. * Improved job naming stability while respecting Kubernetes naming limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6db7260b2a |
feat(kubernetes)!: remove FluxCD addon (#3379)
## What this PR does The Kubernetes app's optional `addons.fluxcd` toggle deployed a Flux Operator + FluxInstance into tenant clusters, served by two vendored Helm charts (`packages/system/fluxcd-operator`, `packages/system/fluxcd`). Those charts are **AGPL-3.0** — the last AGPL content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not depend on them: it runs classic Flux via `internal/fluxinstall`, and migration 21 already removed the Flux Operator from it. The addon defaulted to `enabled: false`. Rather than maintain an AGPL payload for an unused feature, this PR removes it entirely as a breaking change: - Delete both system packages and their `platform` source entries. - Remove the `addons.fluxcd` HelmReleases, values, Go types and generated schema/README from the Kubernetes app. - Add **migration 54**: for any tenant that had the addon enabled, the FluxCD HelmReleases in the management cluster are **suspended, stripped of their Flux finalizer and deleted**, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running inside the tenant keeps running untouched (mirrors the orphan idiom of migration 29). It runs as a `pre-upgrade` hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. After the change the only `affero`/`AGPL` matches under `packages/` are comments in `packages/system/monitoring` explaining why Grafana is not rebuilt — no AGPL payload ships. **Verification:** the Kubernetes app helm-unittests (181) and platform tests (88) pass; `make generate` was run with the CI-pinned `cozyvalues-gen v1.6.0` (schema, README, Go types, CRD); `migrations-target-check` and the `cozystack-version-stamp` bats pass; the `api/apps/v1alpha1` module builds. ### Downstream repositories This removes a user-facing addon and drops the `addons.fluxcd` field from the Kubernetes app's `values.schema.json`. Boxes are left unticked pending follow-ups (not opened from this PR): - **terraform-provider-cozystack** — models the Kubernetes app schema; the removed `addons.fluxcd` field should be checked/dropped there. - **website** — any hand-written mention of the FluxCD addon in the tenant Kubernetes docs should be removed (the managed-app reference regenerates from the README). ### Release note ```release-note feat(kubernetes)!: remove the FluxCD addon (`addons.fluxcd`). It relied on AGPL-3.0 vendored charts and was disabled by default; tenant clusters that had enabled it keep their running Flux release via an automatic upgrade migration. To remove that orphaned Flux from such a tenant, uninstall its `flux-operator`/`flux-instance` Helm releases inside the tenant cluster by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Breaking Changes** * Removed FluxCD addon support from tenant cluster configuration, including the Kubernetes tenant CRD schema, chart values schema, defaults, and published “Cluster Addons” parameters. * FluxCD-related Helm manifests are no longer rendered as part of the Kubernetes package. * FluxCD configuration is no longer accepted for new or updated tenant clusters. * **Migration** * During upgrade, existing FluxCD Helm releases are cleaned up automatically by suspending them, clearing finalizers, and deleting the management-side releases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
851a86dcf1 |
feat(platform): orphan tenant FluxCD HelmReleases on upgrade
Add migration 54 for the removed addons.fluxcd feature. For every tenant that had it enabled, the FluxCD HelmReleases in the management cluster are suspended, stripped of their Flux finalizer and deleted, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running in the tenant keeps running untouched. Runs as a pre-upgrade hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. Mirrors the orphan idiom of migration 29. Bumps migrations.targetVersion to 55. Fail closed. Dropping the finalizer is the only thing that keeps the delete inert, and migrations never re-run, so the script runs under `set -euo pipefail` with no `|| true` on the critical path and gates the delete on the finalizer actually being gone. The verify read uses `--ignore-not-found` rather than `2>/dev/null || true`, so a failed read aborts instead of reading as an empty finalizer list that passes the gate on an object whose state was never checked. A migration that stops is recoverable; one that half-orphans a fleet is not. Cover it with a docker + fake-kubectl unit test running the real script by path inside the migrations image's own alpine base (busybox ash, jq baked on), so it exercises the shipped interpreter and load-bearing pipefail. The suite pins ordering and selection (only the addon HelmReleases are touched, each suspend -> finalizer-drop -> delete in order) and the fail-closed paths (a surviving finalizer, a failing verify read, and a failing fleet scan each abort before deleting and before stamping). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
7cca3d75e7 |
feat(kubernetes)!: remove the FluxCD tenant addon
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a Flux Operator + FluxInstance into a tenant cluster. It was served by two vendored charts, packages/system/fluxcd-operator and packages/system/ fluxcd, which were the last AGPL-3.0 content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not use these charts (it runs classic Flux via internal/fluxinstall; migration 21 already removed the operator from it), and the addon defaulted to disabled. Rather than maintain an AGPL payload for an unused feature, drop it entirely: - delete both system packages and their platform source entries - remove the addons.fluxcd HelmReleases, values and generated schema from the Kubernetes app - drop the addons.fluxcd block still submitted by the e2e harness - drop the removed fluxcd package entries from CODEOWNERS BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is removed. Tenant clusters that had it enabled keep their running Flux release; see the companion migration for the upgrade path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
4385990869 |
feat(redis-operator): migrate from archived spotahome to freshworks-oss fork (#3406)
## What this PR does `spotahome/redis-operator` was archived on 2026-06-11 (read-only; last release v1.2.4, 2022). This migrates the operator to the actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5, Apache-2.0). ### Why freshworks-oss It keeps the same `RedisFailover` CRD and API group (`databases.spotahome.com`) — so existing managed Redis is unaffected — and adds a native `spec.engine: Redis|Valkey` switch, letting managed Valkey drop its Redis-compatibility shim image and run the official `valkey/valkey` image directly. ### Changes - Re-vendor the chart from `oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which carries the `engine`-aware CRD. - Build the operator image from freshworks-oss v3.3.5 source. - Re-target the label/annotation-preservation patch onto the freshworks Go module (`github.com/freshworks/redis-operator`). ### Verification - freshworks-oss source + our `labels.diff` patch compiles natively with Go 1.26.3 (`service` and `cmd` packages). - `git apply --check` of the re-targeted patch against v3.3.5 succeeds. - The operator chart renders with the cozystack-registry image override. - The image is built (freshworks-oss v3.3.5 source + our patch, multi-arch amd64/arm64), pushed to `ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in `values.yaml`. - `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats` pass with the pinned digest. ### Note for maintainers `packages/system/redis-operator` is not part of the automated `make build`; its image is built out-of-band and the digest is pinned in `values.yaml`. That image is already built and pinned in this PR; re-run `make -C packages/system/redis-operator image` only if you bump the operator version. ### Follow-up The cozystack/redis-operator fork carries cert-manager TLS work (cozystack/cozystack#2729) that freshworks-oss does not have yet; the plan is to upstream it to freshworks-oss so it lands in the shared maintained fork rather than a cozystack-only one. ### Release note ```release-note feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for injecting additional environment variables into the Redis Operator (including a default operator group id). * Added default configurable labels for the Redis Operator. * **Bug Fixes** * Preserves and merges existing Service labels and annotations during Redis Operator updates. * **Chores** * Updated the Redis Operator and Helm chart to the Freshworks-maintained 3.3.5 release, aligned chart/image sources, and simplified Helm chart retrieval. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
fd8d378b73 |
fix(kubernetes): rotate bootstrap-token Job on either token half; review nits
Fold bootstrapTokenSecret into the Job content hash alongside bootstrapTokenId. talos-secrets.yaml regenerates the two token halves independently, so hashing only the id left a rotated secret no-oping a completed Job and the tenant stuck on the stale bootstrap-token. Also address review feedback: drop a review-round reference from the tracked test, correct a stale hash in a test comment, pin the alternate-image hash case exactly instead of by regex, and fix a comment indent in talos-reconcile-job.yaml. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
035bde1920 |
fix(kubernetes): correct bootstrap-token Job rationale and make it churn-free
Address review of the main-phase bootstrap-token Job. The deadlock the header claimed cannot occur on this release: the kubernetes cozyrd carries helm-install-disable-wait, which cozystack-api maps to Install.DisableWait and Upgrade.DisableWait, so there is no main-resource wait phase in front of the hooks. Rewrite the rationale around the real reason to leave the hook shape: a post-upgrade hook never runs on a release whose upgrade keeps failing, so the token Secret never lands. Fix the stale 'stays a post-install hook' note in talos-reconcile-job.yaml and the 'fails the chart upgrade loudly' comment, which is untrue once Helm no longer waits on the Job. Give the immutable Job a content-hash name suffix over its whole rendered spec, mirroring talos-reconcile: a static name would fail an upgrade that lands while the Job is live (or inside its TTL) with 'spec.template: field is immutable'. Fold the looked-up bootstrapTokenId into the hash so a rotated token also rolls the Job instead of Helm no-oping a completed one. Make the admin-kubeconfig volume optional and gate the Pod on the shared wait-for-kubeconfig init container, so it waits in init (uncounted by backoffLimit) instead of FailedMount at t0. Extend the template tests for the hashed name, its rotation on an image change, and the optional volume + init container. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
542ab899c2 |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs
The bootstrap-token-tenant Job was wired as a Helm post-install/post-upgrade hook. post-* hooks fire only after Helm's wait phase resolves, and the tenant kubernetes release wait blocks on MachineDeployment Ready, which blocks on nodes registering, which needs the kubelet CSR path — for which the very bootstrap-token this Job produces has to already exist in the tenant kube-system. Fresh installs and any HR reconcile that finds MD in a non-Ready state deadlock; HR.upgradeFailures climbs while post-upgrade hooks never get their turn to run. pre-install/pre-upgrade would not help either: they fire before non-hook templates apply, so <release>-admin-kubeconfig (materialised by Kamaji after KamajiControlPlane is applied) is not yet available. Move the ServiceAccount and Job to the main install phase, matching what talos-reconcile already does in this chart. Helm applies them together with the KamajiControlPlane and talos-secrets; the Job Pod stays pending until Kamaji publishes the admin kubeconfig Secret (backoffLimit covers the retry window) and then completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state and does not gate it on MD. Extend the existing template test to pin the absence of helm.sh/hook on both the ServiceAccount and the Job, so this cannot regress silently. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
68d83cb686 |
test(e2e): add a runner-layer fixed-work canary around the node-join wait (#3919)
## What this PR does The diagnostics around the node-join wait already carry counter pairs that report time in the CPU rows, events in the KVM exits, and instantaneous values and running maxima in the files beside those. None of them is a fixed quantity of work, so none can separate a machine that was given fewer turns from a machine that took its turns and got less done with them. This adds a collector that runs two fixed amounts of work on the runner VM and times them: an integer loop and a streaming write, each under a ceiling of its own. It takes one sample before the node-join wait and one after it, on the failing and the passing path alike. Each sample is an absolute reading rather than half a difference, so one alone is still readable against the expected ranges the capture prints, and the pair says whether a slowdown began inside the interval it brackets or was already there going into it. The collector sits outside the three counter pairs that bracket the same wait rather than among them, because it occupies a core while it runs, and a burn placed inside one of those intervals would be charged to the join window it is supposed to measure. A new bats suite of 57 tests covers the collector. The existing node-join guards were extended to price it against the phase budget and to keep the spend order documented in both kubernetes suites in step with the order the block actually runs. On the budget: the sample taken before the wait is bounded at two arms of the canary ceiling plus grace, so it can add up to fifty seconds ahead of the wait, and the guard prices it at exactly that, while the guard's total stays below an honest worst case because collectors it already priced can spend more than the figures it prices them at. This branch was reviewed extensively before it was opened, and what is known to be incomplete is written down rather than left implied: the words describing how each floor sits against its healthy band are marked in the code as written rather than derived, so they are the part that can go stale when a constant moves, and the phase budget residuals are tracked in the issue the comments beside them cite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. The diff is five files: the e2e diagnostics collector library, its two new and extended bats suites, and the two chainsaw suite definitions that document the spend order. The closest call is `cozystack/ccp`, whose skills gate on `hack/package.mk` and `hack/common-envs.mk` and drive `make generate`. Neither file is touched, nothing under `hack/` is moved or renamed, and no make target changes what it does: the new suite is picked up because `BATS_UNIT_FILES` already globs `hack/*.bats`, so the file list grows and the target's contract does not. Nothing else matches. No package is added, renamed or removed, so the website app lists are unaffected. No `values.schema.json`, version enum, default or `ApplicationDefinition` changes, so the Terraform provider is unaffected. `hack/e2e-prepare-cluster.bats` is untouched, so the node prerequisites that `ansible-cozystack` and `talm` restate are unchanged. `hack/update-crd.sh`, the `release.labels` convention and the chart source kinds are untouched, so `external-apps-example` and `cozyhr` are unaffected. No label, annotation or metric name changes, so `cozy-proxy` and `cozystack-telemetry-server` are unaffected. Minimum node and network requirements are unchanged, so `examples` is unaffected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): the node-join diagnostics now time two fixed amounts of work on the runner before and after the wait, so a run that spent its wall clock without completing work is visible in the report. No user-facing change. ``` |
||
|
Aleksei Sviridkin
|
7ec5dff93a |
test(e2e): add a runner-layer fixed-work canary around the node-join wait
The counters the node-join failure path collects are read in whatever unit their source publishes: time in the CPU rows, events in the KVM exits, instantaneous values and running maxima in the files beside those. Not one of them carries a unit of work of its own, so a machine that spent every tick and got a fraction of the work done reads healthy in all of them. Add a collector that carries its own unit of work. It runs two fixed amounts of work at the runner layer and times each: a loop over a handful of integer-valued scalars, which pressure on the shared cache or the memory controller moves little, and a store stream through blocks larger than the last-level cache one core can allocate into, which is dominated by it. Fixed work gives each reading a scale of its own, so a red run can be read with no green one beside it, and the capture states the expected range and where the estimate comes from. Sampled where the three counter pairs already are, before the wait on every run and after it on both paths, but outside their brackets: it occupies a core rather than reading a file, and a burn inside one of those intervals would land in the interval each pair divides by: runner-kernel CPU time charged to the join window, and for the two counting the guests, a core taken away from what they count. Two arms, each under a twenty-second ceiling with the five-second kill grace behind it, so fifty seconds at worst against the 420s diagnostics budget. The budget guard derives that from the ceiling the source declares, and the phase-warning guard names the canary among the collectors that keep running when timeout is missing. An exit shaped like the ceiling is attributed to it only when the whole bound was on the clock: an outside kill produces the same status, and read by status alone it would publish exactly the finding this collector exists to detect. A reading below the range the legend states, and an arm the ceiling stopped before it could produce one, reach the job log from all three reporting sites, each naming where it was taken, the way a missing reading does from the two outside the diagnostics block. On the passing path the report that holds the figure is the artifact nobody downloads. Each floor is printed in the legend it comes from and sits where a bounded arm crosses it before the ceiling stops the arm, because a floor the ceiling reaches first never fires: the arm is cut off before its rate can fall that far, and every alert then arrives as a bound rather than as the slowdown the floor was meant to name. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
899529b386 |
chore(ci): add area/cilium label and map the cilium PR scope to it (#3914)
## What this PR does Adds an `area/cilium` label and routes the `cilium` PR scope to it. Cilium has more than 20 open issues of its own right now (the pinned version and its bumps, the Gateway API implementation, network policy, IPAM next to kube-ovn, LB-IPAM), and `area/networking` no longer narrows a search. #3913 is the tracking issue that uses the label. The label already exists in the repository, created by hand so the open issues could be labelled before this lands. The sync workflow runs with `delete-other-labels: false`, so nothing is at risk in between; this PR makes `labels.yml` the owner of it and updates the scope table in `docs/agents/contributing.md`. The `cilium` scope moves from `area/networking` to `area/cilium`, following the `kamaji` and `keycloak` precedent where the dedicated area replaces the broad one rather than stacking on it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Added a dedicated `area/cilium` label for Cilium-related issues and pull requests. * Updated automated labeling and contribution guidance so Cilium changes are categorized separately from general networking work. * Refined the networking label description to exclude Cilium-related items. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
467e431765 |
chore(ci): add area/cilium label and map the cilium PR scope to it
Cilium has enough open issues of its own (version pins and bumps, the Gateway API implementation, network policy, IPAM next to kube-ovn, LB-IPAM) that area/networking no longer narrows a search. Declare area/cilium in labels.yml, route the cilium scope of PR titles to it instead of area/networking, and update the scope table in the contributing guide. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
IvanHunters
|
cd72d81bb4 |
Merge branch 'main' into feat/kubernetes-nodes-phase2b
Reconcile the phase-2b split with main's configurable worker-health feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/ podCpuRequest), which landed on both charts after this branch forked: - kubernetes (parent): pool rendering stays removed; main's parent worker-pool additions do not apply post-split. - kubernetes-nodes (pool): absorb the feature; keep the directional parent-version skew guard and the immutable cluster/storageClass rules. KubevirtMachineTemplate content hashes are unchanged (render snapshot). - migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime into adopted pools, so adoption neither rolls nor resets a tuned pool. Covered by a new adopt-values bats case. - drop the parent chart's obsolete MHC-timeout suites; the kubernetes-nodes twins cover the feature. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
myasnikovdaniil
|
79287cd9b1 |
docs(release): add the v1.5.4 changelog, and fix v1.6.0's fail-open audit note (#3910)
Adds the changelog for `v1.5.4`, and corrects one still-live instruction in v1.6.0's. The generated changelog is a good inventory and every one of its bullets is kept. What it had no layer for is the operator: no required-actions section, no runnable pre-upgrade checks, and the SeaweedFS 4.31 rename left as three isolated fix bullets that never say what to do about it. That is what is added on top, in the shape v1.6.0's changelog uses. Base widened to `v1.5.2`. v1.5.3 was tagged but its release was never published and its changelog PR was closed unmerged, so `docs/changelogs/v1.5.3.md` exists on no ref and a v1.5.3 base leaves #3212 (persistent EFI/TPM state) and #3194 (filer postgres2 connection pool) documented nowhere at all. Coverage is 82/82 commits in range. Three claims corrected. The slot 45 divergence is a skip rather than an ordering problem: v1.5.4 is the first 1.5.x release stamped `targetVersion: 46`, and `run-migrations.sh` loops `seq CURRENT (TARGET - 1)`, so a cluster that reaches 46 runs slots 46 through 53 on the way to 1.6 and never executes 1.6's own slot 45. Its chart-side half was missing too and is not redundant, because a fresh v1.5.4 install is stamped 46 having never run any slot, so only the chart can reach that population. And the S3 checksum bullet named a barman-cloud plugin path that does not exist on `release-1.5`. The audit re-run warning names v1.6.0 rather than an earlier release, because the script does not exist at v1.5.2 at all. v1.5.4 is the first 1.5.x release to carry it, and it carries the fail-closed version. The second commit edits `docs/changelogs/v1.6.0.md`, which is deliberate rather than an accident of scope. Published changelogs are historical records and are normally left alone, but v1.6.0's upgrade notes still tell operators to run `hack/seaweedfs-naming-audit.sh`, and the copy that shipped in v1.6.0 silences every `kubectl` call with `2>/dev/null`. A timeout or an RBAC denial therefore prints an empty all-clean table indistinguishable from a genuinely clean fleet, and that output gates a runbook step that deletes PVCs. Verified per tag: v1.6.0 carries 11 blanket redirections and no `run_kubectl` wrapper, while v1.6.1, v1.6.2 and v1.5.4 carry 4 and 18, the fail-closed shape from #3436 and its backport #3474. |
||
|
Myasnikov Daniil
|
12e892325a |
docs(release): warn that v1.6.0's naming audit fails open
v1.6.0's own upgrade notes tell operators to run hack/seaweedfs-naming-audit.sh, and the copy that shipped in v1.6.0 silences every kubectl call with 2>/dev/null. A timeout or an RBAC denial therefore prints an empty "all clean" table indistinguishable from a genuinely clean fleet, and that output gates a runbook step that deletes PVCs. Anyone following those notes today can still get a false clean. Verified per tag: v1.6.0 has 11 blanket redirections and no run_kubectl wrapper; v1.6.1, v1.6.2 and v1.5.4 have 4 and 18 respectively, the fail-closed shape from #3436 and its backport #3474. Published changelogs are historical records and are not normally edited. This one is an exception on purpose, because the instruction it carries is still live and still wrong. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
eb8c19d4a1 |
docs(release): enrich the v1.5.4 changelog
The generated changelog is a good inventory but carries no operator-facing layer: no required-actions section, no runnable pre-upgrade checks, and the SeaweedFS 4.31 rename left as three isolated fix bullets that never say what to do about it. Every generated bullet is preserved; this adds to them. Widen the base to v1.5.2. v1.5.3 was tagged but its release was never published and its changelog PR was closed unmerged, so docs/changelogs/ v1.5.3.md exists on no ref and a v1.5.3 base leaves #3212 (persistent EFI/TPM state) and #3194 (filer postgres2 connection pool) documented nowhere at all. Coverage is now 82/82 commits in range. Correct three claims. The slot 45 divergence is a skip rather than an ordering problem: v1.5.4 is the first 1.5.x stamped targetVersion 46, and run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster at 46 runs 46..53 on the way to 1.6 and never executes 1.6's own slot 45. Its chart-side half was missing too, and it is not redundant -- a fresh v1.5.4 install is stamped 46 having never run any slot, so only the chart can reach it. The S3 checksum bullet named a barman-cloud plugin path that does not exist on this branch. The audit re-run warning names v1.6.0, not earlier releases. The script does not exist at v1.5.2; v1.5.4 is the first 1.5.x to carry it and carries the fail-closed version. v1.6.0 shipped the copy that silenced every kubectl failure, so a clean table from it is not evidence. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
0b8837e552 |
fix(telemetry): make the documented opt-out cover both collectors (#3791)
## What this PR does Telemetry is reported by two components — `cozystack-operator` (cluster facts) and `cozystack-controller` (application counts) — but only the operator had a reachable switch. The documented opt-out sets `cozystackOperator.disableTelemetry` on the `cozy-installer` chart. That chart deploys the operator; it cannot reach a package the platform deploys. `packages/system/cozystack-controller` has its own independent `disableTelemetry` key that nothing propagated to, so after an operator had followed the published procedure the controller kept posting `cozy_application_count` every 15 minutes. Nothing failed, and nothing indicated the opt-out was partial. Two changes: - `packages/core/platform` gains a `telemetry.disabled` key and threads it into the `cozystack-controller` component of the `cozystack.cozystack-engine` Package — the same mechanism the bundle already uses for cilium, multus and linstor values. - `packages/core/installer/values.yaml` declares `cozystackOperator.disableTelemetry`. It was only ever referenced from the template, so the `--set` target the documentation names did not appear in the chart's own values. Both keys default to `false`. Nobody who has not opted out sees a behaviour change. The opt-out remains two steps, because the two collectors are installed by two different charts. Collapsing it into one knob would mean the operator writing the value into the platform Package it does not own, which is a design change rather than a fix. `helm unittest` on the platform chart: 137 passed (133 before, plus the four added here). `helm lint` clean on both charts. ### Screenshots Not a UI change. ### Downstream repositories Trigger map walked against the diff. `packages/core/platform/values.yaml` changed, which the map routes to the website's hand-written platform-package table; the same change also makes the published opt-out procedure wrong, so the docs follow-up covers both. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/651 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(telemetry): the documented telemetry opt-out now covers both collectors. Setting `cozystackOperator.disableTelemetry=true` alone left `cozystack-controller` reporting application counts; the platform gains a `telemetry.disabled` key that silences it. Both default to false. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform telemetry opt-out setting, disabled by default. * Added a separate operator telemetry opt-out setting, also disabled by default. * Applying the platform setting now disables telemetry reporting for the relevant controller. * **Bug Fixes** * Ensured telemetry settings are correctly applied across supported platform variants while preserving existing configuration overrides. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
4b50a3ceb4 |
fix(ingress-nginx): pull the protobuf-exporter from the project namespace (#3783)
## What this PR does
Points the `protobuf-exporter` sidecar at the project's registry.
The sidecar was still pulled from `ghcr.io/kvaps/...` while the
controller beside it — built from the same repository, pinned to the
same `v1.11.2` — already came from `ghcr.io/cozystack/...`. The
repository moved to the organisation some time ago and the image
reference did not follow. It is the last thing in the tree that ships a
Cozystack-owned component out of an individual's namespace, which is a
Required item on the Incubation checklist ("All project metadata and
resources are vendor-neutral").
The image is the same bytes: `v1.11.2` was copied into the organisation
namespace with the manifest digest preserved, so `sha256:6d9235a9…` is
unchanged here. Nothing is rebuilt and nothing is upgraded.
The version is deliberately not bumped. `v1.11.5` exists in the
organisation namespace, but the tag tracks the upstream ingress-nginx
version and the controller sits on `v1.11.2`; bumping only the sidecar
would split the pair.
The comment in `hack/promote-retag.sh` loses its `ghcr.io/kvaps/...`
entry, because that path now falls under the bullet below it — a
cozystack-org repository outside `$REGISTRY`
(`ghcr.io/cozystack/cozystack`). Selector behaviour is unchanged;
`hack/promote-retag_test.bats` passes, 11/11.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The two files touched are a package `values.yaml` image reference
and a comment in `hack/promote-retag.sh` — no trigger matches. In
particular the `ccp` trigger is a move, a rename, or a change to what a
make target does; this is neither, and its anchors (`hack/package.mk`,
`hack/common-envs.mk`) are untouched.
- [x] No downstream repository is affected by this change
### Release note
```release-note
fix(ingress-nginx): the protobuf-exporter sidecar is now pulled from ghcr.io/cozystack instead of a personal namespace. Same image and digest; no version change.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the protobuf exporter image source to the new container
registry location.
* Refreshed image promotion guidance to remove an outdated third-party
registry example.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
cozystack-ci[bot]
|
c6238e1544 |
docs: add changelog for v1.5.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Andrei Kvapil
|
3197c48bc5 |
test(migrations): cover migration 54 redis adopt (label+pin, CRD probe fail-closed)
Drive migration 54 end-to-end against a fake kubectl in a jq-enabled build of
the migrations image's alpine base (busybox ash, the production interpreter),
mirroring the docker+fake-kubectl migration-test harness.
Pinned properties:
- an absent CRD stamps 55 without patching anything;
- a failing CRD probe aborts under set -euo pipefail before stamping (a
one-off apiserver blip must not read as "absent" and skip adoption);
- every RedisFailover across namespaces is labelled onto the freshworks
operator-group, while the engine is pinned to redis:6.2.6-alpine only where
the image is empty (both roles for Harbor; sentinel only where redis.image
is preset, which is left untouched; label-only when both are preset);
- a failed adoption patch aborts before stamping so the Job retries.
Auto-discovered by the Makefile hack/*.bats wildcard; no registration needed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
|
||
|
Andrei Kvapil
|
c9ee4f6de8 |
test(platform): pin redis-operator upgradeCRDs=CreateReplace
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
f2bb0db1fe |
docs(redis): point at the freshworks-oss operator fork
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
c7a3320bb4 |
fix(migrations): pin the engine and fail-close the CRD probe in migration 54
Two defects in migration 54's adoption pass: (1) The label landed before the image pins, so the operator adopted a labelled CR with empty image fields and rolled Redis to the fork's 7.2.4 default — rewriting dump.rdb into a format the later 6.2.6 chart pin can no longer load. Pin spec.redis.image/spec.sentinel.image to redis:6.2.6-alpine in the same pass, only where empty. (2) The CRD probe used a bare `get crd >/dev/null 2>&1` that could not tell NotFound from an API blip and stamped anyway; switch to migration 50's `--ignore-not-found -o name` so a real error aborts under set -e. Also self-contained the renumber note and qualified the resource. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
caf9511365 |
fix(platform): CreateReplace the redis-operator CRDs on upgrade
helm-controller skips CRD upgrades by default, so an upgraded cluster would keep the old spotahome schema and prune the fork's spec.engine field on admission. The schema diff is additive, so CreateReplace is safe. Also refresh the harbor e2e wait comment for the freshworks operator. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
3e250ebfde |
test(harbor): assert the operator-group label and pinned engine
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
fde96dd7da |
fix(redis): pin the Redis engine at 6.2.6 in both RedisFailover charts
The freshworks-oss operator's compiled-in default image moved from redis:6.2.6-alpine to 7.2.4; Harbor pinned no image at all and apps/redis left its sentinels unpinned, so an upgrade would move them silently. 7.2.4 rewrites dump.rdb into a format 6.2.6 refuses to load, so a platform rollback would leave Harbor's Redis crash-looping on its own data. Pin redis+sentinel (Harbor) and the sentinel (apps/redis) at the version they run today; bump deliberately and separately. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
0a3a3bfa77 |
fix(migrations): adopt existing RedisFailovers onto the operator group
The freshworks-oss operator watches RedisFailovers through a server-side operator-group label selector, so a CR without the label is silently dropped from its informer. The chart templates now stamp the label, but an existing CR only gets it when its own HelmRelease next upgrades — after the operator rolls — leaving a window with no healing, and a suspended/stuck HelmRelease never gets it at all. Label every RedisFailover in a pre-upgrade migration, ahead of the new operator. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
023134a20d |
fix(harbor): label the Redis failover with the operator group
Harbor provisions its own RedisFailover; the freshworks-oss operator filters by the redis-failover.freshworks.com/operator-group label, so without it Harbor's Redis is never created and harbor-core CrashLoops. Stamp the same group (cozystack) as the operator and the redis app. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
77cb19c8db |
fix(redis): label RedisFailover with the operator group
The freshworks-oss operator filters RedisFailovers by the redis-failover.freshworks.com/operator-group label matching its OPERATOR_GROUP_ID; without the label it ignores the CR and no Redis is provisioned. Stamp the group (cozystack) so the migrated operator adopts the app's clusters. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
0b57261d38 |
fix(redis-operator): set required OPERATOR_GROUP_ID for the freshworks fork
The freshworks-oss fork exits fatal at startup when operator_group_id is unset, unlike the archived spotahome operator. Pin a stable non-empty value via extraEnvVars so the operator boots instead of CrashLooping and blocking the install. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
abb38ab72f |
chore(redis-operator): pin built freshworks v3.3.5 image digest
Build the freshworks-oss v3.3.5 operator image (with our label-preservation patch) and pin its digest in values.yaml, replacing the placeholder tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
fe01ee17bc |
feat(redis-operator): migrate from archived spotahome to freshworks-oss fork
spotahome/redis-operator was archived 2026-06-11 (read-only). Switch the operator to the actively-maintained freshworks-oss/redis-operator fork, which keeps the same RedisFailover CRD and API group (databases.spotahome.com) and adds a native Valkey engine (spec.engine: Redis|Valkey) — so managed Redis is unaffected and managed Valkey no longer needs a Redis-compat shim image. - re-vendor the chart from oci://ghcr.io/freshworks-oss/charts/redis-operator - build the operator image from freshworks-oss v3.3.5 source - re-target the label/annotation-preservation patch onto the freshworks module The operator image is built out-of-band (make -C packages/system/redis-operator image) and its digest pinned in values.yaml; a maintainer must rebuild and pin before merge (tag is a placeholder here). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
myasnikovdaniil
|
f4ab9ec8da |
docs(release): add changelog for v1.6.2 (#3907)
This PR adds the changelog for release `v1.6.2`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.2.md`.
|
||
|
cozystack-ci[bot]
|
be6e66786f |
docs: add changelog for v1.6.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
7194edf625 |
fix(kubernetes): validate MachineHealthCheck condition timeouts at render time (#3901)
## What this PR does Follow-up to #3898, closing two gaps in the render-time duration guard it introduced. Context: #3513. First, the guard now also covers the MachineHealthCheck condition timeouts (`readyUnknownTimeout`, `readyFalseTimeout`) in both the kubernetes and kubernetes-nodes charts. They need it more than the autoscaler annotation does: the CAPI webhook validates `nodeStartupTimeout` with a 30s floor but does not validate `unhealthyConditions` timeouts at all, so a zero or negative value used to apply silently and remediate a Machine the moment its Ready condition flipped, which for a KubeVirt worker means deleting the VM. Second, the guard accepts the canonical form a Go duration serializes back to ("20m0s", "1h0m0s"). `metav1.Duration` prints that shape, so a value copied from `kubectl get -o yaml` output previously failed the render. The first segment must still be positive and later segments may be zero, so every accepted string still parses to a positive duration and the overflow and fraction-to-zero rejections stay. Mechanically, the guard moves into a `kubernetes.positiveDuration` named template in each chart's `_helpers.tpl`, and the error message format is unchanged. Tests cover the canonical form on all three fields and the rejection of unitless, zero, and negative condition timeouts in both charts. The previous test case that passed unitless values through verbatim is reworked into rejections: a unitless value used to fail only at CAPI decode on apply, and a zero or negative one did not fail anywhere. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): reject zero and negative MachineHealthCheck condition timeouts at render time, and accept the canonical Go duration form ("20m0s", "1h0m0s") for the worker join timer values ``` |
||
|
Aleksei Sviridkin
|
3710541278 |
fix(kubernetes): validate MachineHealthCheck condition timeouts at render time
The CAPI webhook validates nodeStartupTimeout with a 30s floor but does
not validate unhealthyConditions timeouts at all, so a zero or negative
readyUnknownTimeout or readyFalseTimeout applied and remediated a
Machine the moment its Ready condition flipped. Route both fields
through the same render-time duration guard as maxNodeProvisionTime,
extracted into a kubernetes.positiveDuration helper in each chart.
The guard now also accepts the canonical form a duration serializes
back to ("20m0s", "1h0m0s"): metav1.Duration prints that shape, so a
value copied from kubectl output previously failed the render. The
first segment must still be positive and later segments may be zero,
which keeps every accepted string a positive duration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
|
||
|
Aleksei Sviridkin
|
b32ca5cbb3 |
chore(kubernetes): drop the force-delete flag that blocks partial cleanup (#3899)
## What this PR does Removes `--force-delete-unregistered-nodes=true` from the tenant cluster-autoscaler Deployment. On the clusterapi provider the flag cannot do what its name promises: the provider's `ForceDeleteNodes` returns `cloudprovider.ErrNotImplemented`, the autoscaler falls back to the ordinary `DeleteNodes` path, and that path enforces the node group's minimum size. The one behavior the flag does change is harmful here: it skips the step that trims a long-unregistered delete batch down to the group's headroom above min size, so a batch larger than the headroom is refused whole where the default would trim it and delete what fits. Context: #3513. A regression test renders the Deployment, proves the args list is live with a positive anchor on `--cloud-provider=clusterapi`, and asserts the flag is absent, so reintroducing it fails the suite. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note chore(kubernetes): drop the cluster-autoscaler force-delete-unregistered-nodes flag; the clusterapi provider does not implement force deletion, and the flag only made batch removal of long-unregistered machines all-or-nothing ``` |
||
|
Aleksei Sviridkin
|
245bca3366 |
chore(kubernetes): drop the force-delete flag that blocks partial cleanup
`--force-delete-unregistered-nodes` reads as an override that lets the autoscaler remove a long-unregistered machine below its node group's min size. On the clusterapi provider it grants nothing of the sort: force deletion is not implemented there, the call falls back to the ordinary delete path, and that path refuses to take a group below its min size. The flag never bought the safety valve its name suggests. What it does buy is a behaviour nobody asked for. With the flag set the autoscaler skips trimming the delete batch down to the group's headroom above min size, and the ordinary delete path then refuses a batch larger than that headroom outright instead of removing what it can. A group of three machines above a min size of one loses two long-unregistered machines with the flag unset, and none with it set. Removing it restores the documented default and takes away a signal that advertised a protection the provider does not offer. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
5fe9450145 |
fix(kubernetes): widen and expose the worker node join timer budgets (#3898)
## What this PR does Worker node join on nested tenants regularly outruns the timer budgets the charts shipped, and the timers then destroy capacity that was about to arrive: the cluster-autoscaler treats a machine as long-unregistered after its built-in 15m default and orders a replacement, and the MachineHealthCheck deletes a Machine that is still legitimately booting. The timer audit in #3513 (https://github.com/cozystack/cozystack/issues/3513#issuecomment-5329896688) walked every timer on the join path against the cozyreport artifact of a red run; this PR sizes the budgets from those measurements and makes the hardcoded ones configurable. Context: #3513. Three changes, one commit each: - Every worker MachineDeployment in both the kubernetes and kubernetes-nodes charts now carries the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, rendered from a new `maxNodeProvisionTime` value defaulting to "30m". The autoscaler starts this clock when the machine enters its node-instances cache with a real providerID, not at Machine creation, and on a value it cannot parse it silently falls back to its 15m default rather than failing, so the template validates the value at render time and rejects anything that does not parse to a positive duration. - The MachineHealthCheck `unhealthyConditions` timeouts become values (`readyUnknownTimeout` default "5m", `readyFalseTimeout` default "10m"; nested under `nodeHealthCheck` in the kubernetes chart, flat in kubernetes-nodes) instead of the hardcoded 30s and 300s. 30 seconds of `Ready=Unknown` deletes a worker whose guest was merely starved of CPU, and 300 seconds of `Ready=False` is less than a CNI rollout takes on a loaded host. - The `nodeStartupTimeout` default rises from "10m" to "20m". The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets up to 20m for the disk image import and the VM to come up, and then up to 20m again for the guest boot and kubelet registration. The `maxUnhealthy` docs now also spell out the round-down trap in small groups: at "50%" a group of one never remediates. The budgets are coherent with the measured phases: in the audited runs a healthy disk import took 2m22s and import plus VM start 2m53s, while the stuck runs spent over 17 minutes holding a running VM with no Node, so 20m per MachineHealthCheck leg and 30m on the autoscaler clock absorb that shape of slowness without letting a genuinely dead machine linger much longer. New helm-unittest suites cover the annotation rendering, the render-time duration guard (including the values the guard exists for: unitless numbers, unknown units, zero, negatives, parser overflow), and the MachineHealthCheck timeout rendering in both charts. The render-parity test no longer pins the MachineHealthCheck values on either side and instead compares the two charts' defaults, so a future drift between parent and child defaults fails the suite. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: TODO-LEAD-FILE-ISSUE (suggested title: "Kubernetes/KubernetesNodes: model the new readyUnknownTimeout, readyFalseTimeout and maxNodeProvisionTime fields (upstream cozystack#NNNN)"; the provider models node_health_check for both resources and gains no typed access to the three new fields until it learns them) - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): widen the worker node join timer budgets and make them configurable: the cluster-autoscaler maxNodeProvisionTime is now set per MachineDeployment (default 30m), the MachineHealthCheck Ready=Unknown and Ready=False timeouts are values (defaults 5m and 10m), and nodeStartupTimeout defaults to 20m ``` |
||
|
Aleksei Sviridkin
|
ddb15ebc6a |
fix(kubernetes): widen the worker node startup budget to 20m
nodeStartupTimeout is the window a Machine has to produce a Node at all, measured from the later of the Machine's creation and its InfrastructureReady transition. What has to fit inside it is the guest boot and the kubelet registering with the tenant apiserver, and on a host under load neither is quick. At 10m a worker that was still booting was deleted before it ever joined, and its replacement started the same race from the beginning. Also record what a percentage maxUnhealthy does at small group sizes, because the interaction is not what the field name suggests: the value is scaled against the machines the check covers and rounded down, and remediation proceeds only while the unhealthy count is at or below the result. At the default "50%" a group of one never remediates, and a group of two remediates one unhealthy machine but not two. The golden-parity gate pinned the child chart's MachineHealthCheck values to the parent's defaults as inputs, which made a default moved on both sides fail there for the wrong reason. Drop the pins so the comparison is between the two charts' defaults, which is the contract it exists to hold, and pin the defaults themselves in each chart's own unit tests instead. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
33701800d6 |
fix(kubernetes): make the worker MachineHealthCheck timeouts configurable
The two unhealthyConditions timeouts were fixed in the template at 30s for `Ready=Unknown` and 300s for `Ready=False`, and both are too tight for a worker joining on a loaded host. `Ready=Unknown` is what the node lifecycle controller writes when a kubelet stops reporting, on a grace period of its own, so 30s on top of that deleted the VM of a guest that had merely stopped being scheduled for a stretch, before it had a chance to answer again. And `Ready=False` is the condition a registered Node holds until its CNI is up, so 300s was the entire budget a first boot got for the CNI rollout to reach it -- a worker deleted mid-rollout takes the rollout with it and its replacement starts over. Move both to values, defaulting to 5m and 10m. The remediation of a worker that is genuinely dead is correspondingly later, which costs less than it appears to: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on either of these timers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
83f0eef645 |
fix(kubernetes): give the autoscaler the real worker join time
The cluster-autoscaler treats a machine as long-unregistered once its own max-node-provision-time elapses, and a worker that has not registered by then stops counting towards its node group's upcoming capacity. On a host slow enough that the Talos image import, the guest boot and the CNI rollout do not fit inside that window, the timer fires on machines that were still joining: the group reads as short of capacity, so the autoscaler asks for replacements, and the extra provisioning load makes the join it was waiting for slower still. Render the per-node-group override the autoscaler reads off the MachineDeployment, sourced from a new maxNodeProvisionTime value that defaults to 30m. The duration is checked at render time because the autoscaler discards one it cannot parse and falls back to its own default without failing, and because the values schema types the field as a string and so accepts every malformed duration. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
59cc7d7114 |
test(e2e): anchor the phase-warning guard on its own unique phrase (#3897)
The guard in the CPU throttle suite finds the missing-timeout phase warning by the first match of a phrase that stopped being unique when the bracket-pair captures started writing their own [bounds] notes with the same words. The first match is now a note, the guard reads it as the warning, and the unit gate fails on every run, blocking all PRs. Same fix the node-join copy of this guard already carries: anchor on the phrase unique to the warning itself. All fifteen suites that read the shared library as text pass locally with the change. |
||
|
Aleksei Sviridkin
|
ca058ad964 |
test(e2e): anchor the phase-warning guard on its own unique phrase
The guard located the missing-timeout warning by the first match of "timeout is not on PATH" in the shared library. The bracket-pair captures now write [bounds] notes carrying that same phrase earlier in the file, so the guard read a note as the warning and failed on every run, holding the unit gate red for all pull requests. The node-join copy of this guard was moved to the warning's own phrase for the same reason; this aligns the remaining copy. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
ca9a9300bb |
chore(e2e): ignore the artifacts the sandbox bringup writes to the repo root (#3891)
## What this PR does Adds a .gitignore block for the files `hack/e2e-prepare-cluster.bats` writes to the repository root when the e2e sandbox is brought up outside CI: the Talos secrets bundle, the rendered machine configs, the kubeconfigs, the boot image and the per-VM `srv1..3/` directories. None of these was matched by anything in .gitignore (`git check-ignore` answers "not ignored" for every one), so in a tree that once ran the bringup a broad `git add` stages a credentials bundle -- `secrets.yaml` holds the cluster's private keys. This came one pre-push check away from landing in a PR once, which is what moved it from a hypothetical to a standing trap worth closing. All patterns are anchored to the repository root, so files with the same names elsewhere in the tree stay visible to git. |
||
|
Aleksei Sviridkin
|
6ac97dba75 |
chore(e2e): ignore the artifacts the sandbox bringup writes to the repo root
hack/e2e-prepare-cluster.bats leaves the Talos secrets bundle, the rendered machine configs, the kubeconfigs, the boot image and the per-VM directories in the repository root, and nothing in .gitignore matched any of them. In a tree that once ran the bringup, a broad git add stages a credentials bundle; this was caught in review once and is a standing trap rather than a one-off. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
974c280699 |
test(e2e): price the layers above the sandbox on the node-join window (#3890)
## What this PR does Adds two readings to the tenant node-join path, taken on either side of the same wait the KVM counters already bracket: the runner VM kernel's own `/proc/stat`, and the per-thread CPU time of the QEMU processes the sandbox VMs run as. Every CPU reading on this failure path has a subject inside the sandbox: a cgroup, a tenant worker's threads, a sandbox node's own `/proc/stat`. Each of them can say a guest was given its ticks while getting no work done, and none can say what the layers above spent. That is where this suite's own tax is paid. The sandbox nodes' guest time is charged to the runner kernel as user time, and whatever the runner VM loses to the machine hosting it is charged to nobody the sandbox can see, so a node reporting `steal 0` and a node whose hypervisor was itself waiting look identical from inside. `/proc/stat` is not namespaced, so reading it from the sandbox container reads the runner VM's kernel. That is the only place steal on that layer appears, and it is read in one direction only: a steal that climbs proves this runner VM was preempted, while a zero proves nothing, because the column is filled only where the hypervisor exposes a paravirt steal clock and nothing in the capture observes whether this one does. The sandbox nodes a layer down can make the stronger claim, since this repository starts them with `accel=kvm`; nobody here starts the runner VM, so for this lane the answer is unknown until somebody reads a non-zero. The sandbox VMs are QEMU processes in the same PID namespace as the shell driving the suite, so their threads are a local read too. What that adds is not "which node is busy" on its own, since `cozy_capture_sandbox_node_cpu_time` already reads each node's own `/proc/stat` by name; it is that split *on the join window*, since the existing collector samples twice across a twelve-second interval inside the diagnostics block, after the join has already failed. Beyond the window it adds the host-side accounting of each vCPU thread, what the runner actually granted it, which no in-guest counter can report, and QEMU's non-vCPU IO and worker threads, which no in-guest counter sees either. It also names which node each process is, read from the `srv<N>/qemu.pid` files the bringup already writes: a container pid identifies nothing, and whether the starving guest is the control-plane node or a worker-carrying one is the discriminating question. With no usable pid file the capture stays anonymous and says so, as its KVM sibling does; the pid reads themselves are bounded like every other read here, since a pid file that never yields a byte must not hold the capture. Both are taken at the two points the KVM counters already use, for the same reason: the values are cumulative, so a rate needs a pair, and the interval worth measuring is the window the guest is failing to boot in rather than a few seconds after the deadline has expired. The thread reading runs the probe the tenant worker capture already uses rather than a copy of it, because that probe already answers the two outcomes that matter here and already names the three ways it can come up short, and a second copy would drift from it exactly at those failure paths. Each capture carries the layer it describes in its own legend. Three kernels in this report publish a `/proc/stat` and two captures now read QEMU threads, one layer apart in each direction, so a reader who takes one for the other is comparing a hypervisor against its own guest. The runner-kernel legend also carries the trap specific to that layer: nearly all of its user time IS guest time, because running the sandbox VMs is what that kernel does, so adding the two columns double-counts the whole run. Cost and blast radius: two bounded local reads per sample, no cluster access, nothing added to any diagnostics-phase budget window. The failure-path calls sit behind the same reasoning as the KVM reading, which the phase-budget guard in `hack/run-kubernetes-node-join_test.bats` now prices with a cost arm for each; the guards that enumerate collectors (the missing-`timeout` warning and the documented spend order) carry both new names. Tests are in `hack/run-kubernetes-runner-cpu_test.bats`, 44 cases, green under both runner forms. They cover the readings themselves, every legend claim a reader depends on, the bounded read and its knobs, the sample-number split, each way a capture can come up empty (a file with no `cpu` row, a container with no QEMU, a proc mount that yields nothing, a QEMU named and then gone), the routing of a message from a read that succeeded against one that failed, the per-node attribution and each way it can be incomplete, and placement guards of their own: both pairs must bracket the wait on both paths, nothing may be inserted between the three second readings, and each collector must report a shortfall to the job log in its own words rather than only to the report, since one call site is the passing path where the artifact is the one nobody downloads. Two guards exist because a legend can fail without any number being wrong: one asserts that the parse-from-the-last-parenthesis rule the legend states actually works on the lines the capture holds, and one that every bracketed legend line lands as a single line, since an apostrophe inside one of these single-quoted strings shreds it into one word per line while still passing any short grep. No documentation change: the KVM pair this mirrors is not described in `docs/agents/e2e-testing.md` either, and its documentation lives in the legends the captures carry. Context: #3513 ### Screenshots No UI changes. ### Downstream repositories One line of the bringup changes too: `hack/e2e-prepare-cluster.bats` now starts each sandbox QEMU with `-name guest=srv<N>,debug-threads=on`. QEMU names vCPU threads (`CPU N/KVM`) only when asked to, so without the flag every thread reports the bare process name and the vCPU/IO split this capture documents would not exist to read on this lane; a test pins the flag to the legend that relies on it. Walked the trigger map in `docs/agents/contributing.md` against the diff. Of the `hack/` triggers it names, `hack/e2e-prepare-cluster.bats` (ansible-cozystack, talm) is touched: the touched lines are the CI sandbox QEMU flags, which neither downstream repository restates -- both provision real machines, not the e2e sandbox. `hack/package.mk`, `hack/update-crd.sh` and `hack/upload-assets.sh` are untouched. The rest of the change is confined to the Chainsaw kubernetes suites' shared library and its unit tests, which no downstream repository restates. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): a tenant node-join failure now records the runner kernel's own CPU time and the sandbox VMs' per-thread QEMU CPU time on either side of the join wait, so steal on the layer above the sandbox and an uneven split across the three nodes are both visible in cozyreport.tgz ``` |
||
|
Aleksei Sviridkin
|
b1b8cbecc2 |
test(e2e): price the layers above the sandbox on the node-join window
Every CPU reading on this failure path has a subject inside the sandbox: a cgroup, a tenant worker's threads, a sandbox node's own /proc/stat. Each of them can say a guest was given its ticks while getting no work done, and none can say what the layers above spent. That is where this suite's own tax is paid: the sandbox nodes' guest time is charged to the runner kernel as user time, and whatever the runner VM loses to the machine hosting it is charged to nobody the sandbox can see. Two readings close that. /proc/stat is not namespaced, so reading it from the sandbox container reads the runner VM's kernel -- the only place steal on that layer appears, which is what separates a slow suite from a machine sharing a core with somebody else. The sandbox VMs are QEMU processes in this container's own PID namespace, so their threads are a local read too, and they say which node spent the kernel's user time and whether the three spent it evenly. Both are taken at the same two points as the KVM counters and for the same reason: the values are cumulative, so a rate needs a pair, and the interval worth measuring is the window the guest is failing to boot in. The thread reading runs the probe the worker capture already uses rather than a copy of it, since a second copy would drift exactly at the failure paths. Each carries the layer it describes in its own legend, because three kernels in this report publish a /proc/stat and two captures read QEMU threads, one layer apart in each direction -- and a reader who takes one for the other compares a hypervisor against its own guest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
c5e175a88c |
test(e2e): record the runner VM shape and region in the e2e report (#3889)
## What this PR does Records which machine each e2e run landed on, in the job log and in `cozyreport.tgz` under `sandbox-host/runner-identity.txt`: the OCI shape, both region keys, the availability and fault domain, the numbers of the `shapeConfig` sizing object, and the runner VM's own kernel release, online CPU count and processor model. An e2e lane asks for a runner class, not for a machine. A flex shape is filled from whatever capacity the pool has at the time, so one lane can be served by two processor generations, and the region is a preference with a fallback. None of that reached the report, so a red run and a green one could not be split by the hardware they ran on. That is the first thing worth knowing about a flake that some runs have and others do not, and today the answer is nowhere in the artifact. `hack/e2e-runner-identity.sh` takes the read on the runner VM, before the sandbox container is created, because the instance metadata service answers on a link-local address the container has no guaranteed route to. An "unreachable" answered from inside the container would look exactly like a capture that had tried. The call sits in `packages/core/testing/Makefile`'s `apply` target rather than in a workflow step: every e2e lane reaches the sandbox through `make prepare-env`, so one call covers all of them, on the passing path and the failing one, with no per-workflow step to keep in step. `hack/cozyreport.sh` folds the file into the report from inside the container, without going near the endpoint itself. What it holds to: - **The root instance document is never requested.** The keys are fetched one at a time. That document carries the instance and compartment OCIDs, the image OCID, the display name, the hostname and the user-supplied metadata bag, and the report is a public artifact, so asking per key makes the privacy property something the wire settles rather than something a filter has to be trusted for. It also matches where the shape sizing is actually published: `shapeConfig` has its own subpath and is not in the root document at all, so a reader hunting it inside that document would have reported the size of every machine as unknown and blamed the service for it. From the sizing object only the numbers are kept, and only under keys shaped and sized like field names, which holds for keys nobody has seen yet since a size is a number, an identifier is a string, and a key is printed into the record exactly as a value is; that object is refused whole unless it arrives as one complete JSON object, so a truncated answer cannot be read in part and published as a sizing record; and a per-key answer that is not one scalar is named as such rather than printed, because a service answering a whole document to a key request is the one way this design could still publish what it never asked for. - **Never fatal, and never slow.** This runs inside `Prepare environment`, the one retried step in the job, so a diagnostic that failed it would cost a whole run. Every read is taken through a wall-clock ceiling and every failure is written into the record instead: a runner that is not an OCI instance leaves an `imds unreachable` line and an exit status of 0, and a wedged `lscpu` is cut off and named rather than holding the step. Where `timeout` is not on PATH the local reads run unbounded rather than not at all, and the record says so, which is the same trade `hack/cozyreport.sh` makes for its own. Exiting zero is not enough on its own, since a read that never returns never reaches that exit. The first key doubles as the probe and is held to answering the way this service does, so a machine that is not an OCI instance pays one ceiling rather than one per key whether the address is silent or something else is answering on it, and either way the run leaves an `imds unreachable` line to grep for. - **One filename either way.** When there is no record, the report writes that same file holding the reason there is none, because an unidentified runner and an unremarkable one otherwise leave the same absence. The record also goes to the job log, and that copy is what covers the one path where the artifact cannot: a run whose sandbox never came up produces no `cozyreport.tgz` at all, so the log is where the record lives. Tests are in `hack/runner-identity.bats`, 52 cases, green under both runner forms. They cover the sizing-object number reader (a renamed key, non-numeric fields counted rather than printed, a nested value, an escaped quote inside a string value, input that is not JSON), the scalar guard (including a whole document answered to a key request), the ceiling knob including the zero that would remove the bound outright, and the running script: which paths are requested and that the root document is not among them, the probe short-circuit, a single key failing without the service being called unreachable, a partial transfer of the sizing object, the bounded local reads and the cut-off arm, a write that would never return, the record reaching both sinks, and both arms of the report side. Verified against mawk and dash as well as the local defaults, since the runner image is neither gawk nor bash. Context: #3513 ### Screenshots No UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. The `hack/` triggers it names are `hack/e2e-prepare-cluster.bats` (ansible-cozystack, talm), `hack/package.mk` and `hack/update-crd.sh` (ccp, external-apps-example) and `hack/upload-assets.sh` (website); none of those is touched. The new script and its suite are new files with no counterpart anywhere, `hack/cozyreport.sh` is not restated downstream, and the `packages/core/testing/Makefile` edit adds a diagnostic to the e2e sandbox bring-up without changing what `make prepare-env` does for a caller. The docs change is a bullet in `docs/agents/`, which no downstream repository mirrors. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): every e2e run now records the runner VM it landed on (OCI shape, both region keys, availability and fault domain, shape sizing numbers, kernel release, CPU count and processor model) in the job log and in cozyreport.tgz, so red and green runs can be compared by hardware ``` |