mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-10 14:34:58 +09:00
c90a7b07dba3122ddcd490a2abb7eb19be485aa3
5661
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
c90a7b07db |
fix(kubeovn-webhook): add NetworkPolicy and mTLS support
Address unauthenticated access vulnerability (GHSA-g883-q79m-8225) by: - Adding NetworkPolicy to restrict ingress to kube-system namespace only - Adding optional mTLS support via --client-ca-file flag for client certificate verification https://claude.ai/code/session_013DRdaUJMYxonsjVpWxJSq3 Signed-off-by: Claude <noreply@anthropic.com> |
||
|
Aleksei Sviridkin
|
9ccb133b66 |
fix(ci): de-escalate select-e2e and name every escalation (#3817)
## Why
I replayed the 150 most recently merged pull requests through
`hack/select-e2e.sh` with every escalation branch instrumented. 118 of
them (78.7%) ran the full 21-suite Chainsaw run, and the causes, counted
once per escalating PR with the count where that cause was the only one
in brackets, were `FULL_PATTERN` 88 [68], `NO_SUITE_FOR_GROUP` 24 [24],
`CHAINSAW_SHARED` 18 [1] and `UNCLASSIFIED` 7 [4]. `YQ_BROKEN`,
`NO_GRAPH_OWNER` and `BACKSTOP_EMPTY` never fired.
Two problems come out of that. Most of the escalation is real but
unnecessary. Whole classes of path that provably cannot affect a
Chainsaw suite were escalating because no rule claimed them. And the
commonest cause of a full run was absent from the log, so "why did this
pull request run everything" could only be answered by re-deriving the
selection by hand.
## What changes
**Every escalation now names its cause on stderr.** Seven branches could
reach the full suite and four did it in silence, `full_suite_pattern`
among them, the commonest cause by a wide margin, so the usual answer
was the one the log never carried. The lines go to stderr and must stay
there: stdout is the suite list and both e2e lanes parse it, so a reason
line on stdout would be read as a suite name. A unit test asserts every
message, verified by muting each of the eleven in turn and confirming
the file goes red for all of them. A reason that regresses to silence
changes no selection and is otherwise invisible.
**`cozystack.etcd-application` gains its missing `dependsOn:
cozystack.etcd-operator`.** `packages/extra/etcd` renders `kind:
EtcdCluster` from `etcd-operator.cozystack.io/v1alpha2`, and this was
the only operator-backed application source in the tree with no edge to
its operator. It is a latent install bug as much as a selection gap:
without the edge `etcd-rd` registers the ApplicationDefinition as soon
as the engine is up, so a tenant can create an `Etcd` before the
operator exists and its HelmRelease fails on `no matches for kind
"EtcdCluster"`, and `hack/select-install.sh`'s forward closure for the
`etcd` suite omitted the operator for the same reason. On the selection
side `cozystack.etcd-operator` reached no runnable suite, so every
change to the operator ran all 21.
**`vm-disk-application` maps to the `vminstance` suite.** It resolved to
a `vm-disk` suite that does not exist, `intersect_suites()` dropped the
name, and the source reached nothing runnable. Neither did
`cozystack.kubevirt-cdi` above it, whose only other dependents are
`vm-default-images` and `vm-disk-application` itself, so every CDI
change ran all 21. The coverage was there and the table did not know it:
the `vminstance` suite creates a `VMDisk` and asserts the `DataVolume`
behind it (`hack/e2e-chainsaw/vminstance/vmdisk.yaml`,
`vmdisk-vmi.yaml`). Same defect class as the etcd edge above, on the
mapping side rather than the graph side, and with no production blast
radius of its own, since `src_to_suites` is read by the selector and by
nothing else.
**`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk.** Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs, which `kubevirt-cdi` states in its own
source ("Depend on cozystack-basics so the target namespace exists
first"), and that is install ordering rather than behaviour. The damage
runs opposite to the engine's: the engine fans one change out to every
app, basics narrows instead. It reaches no suite today so it escalates,
but it sits upstream of `kubevirt-cdi`, so the first suite to land under
CDI silently converts the platform's namespace-and-policy package from
the full run to that one suite. Not hypothetical — on #3426 enabling the
site-router suite takes `cozystack-basics` from 22 suites to
`site-router` alone, and nothing in the output says coverage was lost.
Dropping the reverse edges keeps the package reachable, stops it
propagating, and leaves a change to it running everything through the
per-path escalation.
**`hack/e2e-apps/<name>.bats` maps to the `<name>` suite**, off the
basename, exactly as the per-suite rule takes the name off a
`hack/e2e-chainsaw/<app>/` directory. Deliberately mapped rather than
marked inert: what remains there is wired to nothing after the Chainsaw
migration, and inert would bake that orphan status into the rule and go
quietly wrong the day a lane runs those files again. The basename is
membership-tested against the suite list on the spot, so an unmatched
one escalates there and then rather than having the verdict deferred to
the final intersection, where the rest of the diff would decide it.
**`packages/tests/` and three `.gitattributes` join
`inert_config_pattern`,** which is what the script's own header says to
do with a genuinely inert path instead of widening the fall-through.
`packages/tests/` is a helm-unittest fixture chart, and changing a test
*of* `cozy-lib` does not change `cozy-lib`, no PackageSource lists those
paths as a component, and nothing installs them, while a change to the
library itself still escalates through `packages/library/`. The
`.gitattributes` entries are enumerated rather than matched by filename,
because the justification is what those files contain (only
`linguist-generated` markers) and the name does not carry it:
`.gitattributes` can also set `filter`, `eol`, `working-tree-encoding`
and `export-subst`, each of which changes what lands in the working tree
and therefore what gets built.
**`full_suite_pattern` escalates only `hack/e2e-*.bats`, not every
`hack/*.bats`.** The root `Makefile` is the authority on the split and
draws it at exactly that prefix, `BATS_UNIT_FILES := $(filter-out
hack/e2e-%.bats,$(wildcard hack/*.bats))`, so the 60 files it keeps are
the unit lane and the e2e sandbox runs none of them, while the three it
filters out are what `packages/core/testing`'s recipes execute and those
still escalate. Being inert here does not leave them untested: `make
unit-tests` is gated on the `plan` job's `code` output, which
`pull-requests.yaml` computes as "any changed path outside `docs/`" and
never from `select-e2e.sh`, so a bats-only pull request still runs the
whole unit lane, plus install and the OpenAPI tests, since the `e2e` job
reads the same output rather than the selection.
## Measured effect
The before-and-after comparison below is a second measurement over a
slightly different population, stated separately because the two are not
interchangeable: it replays the last 150 first-parent merge commits on
`main` (`git diff --name-only $sha^1 $sha`) through the base selector
with base sources versus this branch with its own, where the histogram
above walks the 150 merged pull requests as GitHub lists them. The
populations overlap heavily and the direction is the same, and the
totals differ by two commits.
| | full suite | scoped | nothing |
|---|---|---|---|
| before | 120 | 20 | 10 |
| after | 106 | 25 | 19 |
Full-suite rate 80.0% → 70.7%, verdict changed on 15 commits. Fourteen
de-escalated and I read every one of their file lists: they are
unit-lane bats files, non-e2e workflows, `docs/`, `CODEOWNERS`,
helm-unittest fixtures and linguist markers. The fifteenth gets one
suite *wider*: a `cert-manager` change now also selects `etcd`, which is
correct, since `cert-manager` is a dependency of `etcd-operator` and the
etcd app now genuinely reaches it through the new edge.
That replay predates the two reachability rules, and replaying the same
150 first-parent merges across those two on their own — the selector at
`73bc4c0ae` against the selector at `adf231513`, same sources on both
sides — changes no verdict at all. Two of the 150 touch
`cozystack-basics` and both ran the full suite before and after, which
is the hub exclusion keeping something true rather than failing to do
anything, and none of the 150 touch `kubevirt-cdi` or `vm-disk`, so the
mapping shows up only in a direct selection.
A few individual selections, before → after:
| changed files | before | after |
|---|---|---|
| `hack/select-e2e_test.bats` | 21 suites | *nothing* |
| `hack/select-e2e_test.bats` +
`packages/apps/redis/templates/redis.yaml` | 21 suites | `redis` |
| `hack/e2e-install-cozystack.bats` | 21 suites | 21 suites |
| `packages/tests/cozy-lib-tests/**` | 21 suites | *nothing* |
| `packages/system/.gitattributes` | 21 suites | *nothing* |
| `hack/e2e-apps/postgres.bats` | 21 suites | `postgres` |
| `packages/system/etcd-operator/values.yaml` | 21 suites | `etcd` |
| `packages/system/kubevirt-cdi/values.yaml` | 21 suites | `vminstance`
|
| `packages/apps/vm-disk/values.yaml` | 21 suites | `vminstance` |
| `packages/system/cozystack-basics/values.yaml` | 21 suites | 21 suites
|
## Blast radius of the `dependsOn` edge
A `dependsOn` on a PackageSource is an install-ordering edge in
production, not only a test-selection hint, so this is the part worth
reviewing hardest. Both Packages are emitted by
`packages/core/platform/templates/bundles/system.yaml` under the single
`bundles.system.enabled` guard with no intervening conditional between
them, in every variant, so no supported configuration has the app
without the operator. `hack/select-install.sh --validate` reports the
graph still has no cycle and nothing dangling. The one new exposure is
an admin who lists `cozystack.etcd-operator` in
`bundles.disabledPackages` while keeping the app, which now leaves it
`DependenciesNotReady`. That is the same exposure postgres, mariadb,
kafka and redis already carry, with `Package.spec.ignoreDependencies` as
the escape hatch. On upgrade `etcd-rd` waits for the operator's two
HelmReleases, and an unhealthy `etcd-operator` already means broken etcd
apps.
## Testing
Every rule added or changed has a test in `hack/select-e2e_test.bats`,
the install-side change has one in `hack/select-install_test.bats`, and
the new graph edge has a helm-unittest guard at
`packages/core/platform/tests/sources_etcd_application_dependson_test.yaml`.
The round-trip test that walks every suite through both mapping tables
stays green, `vminstance` included now that two sources map onto it.
The two reachability rules were each checked by removing the rule and
confirming its own test reddens: without the mapping the CDI test falls
back to the full-suite escalation, and without the hub exclusion
`cozystack-basics` selects `redis vminstance` instead of everything. The
hub test has to seed its own hazard, since no suite sits downstream of
basics in this tree yet, so it gives `redis-application` an edge onto
basics in a copied sources directory and pins that a redis change still
selects redis, or the assertion would pass against a graph that resolved
nothing.
`make bats-unit-tests` aborts on the first failing file (`for f in …; do
… || exit 1; done`), and `hack/ghcr-mirror_test.bats` fails on `main`
today and sorts before `select-e2e_test.bats`, so the aggregate target
never reaches these tests. I ran the 60 unit files individually instead:
59 pass, and the one failure is that pre-existing one, byte-identical to
the base commit and referencing nothing this branch touches. Also green:
`make helm-unit-tests`, `make rd-presets-check migrations-target-check
test-check-readiness`, `hack/select-install.sh --validate`, and
`pre-commit run --all-files` leaving a clean tree.
## Two things a reviewer should know, neither addressed here
`assert_selection` in `hack/select-e2e_test.bats` aborts the whole file
with `exit 1` rather than failing a single test. That is the only thing
that works under `hack/cozytest.sh`, which provides no real `run`,
`$status` or `skip`, and it is the idiom the existing tests in that file
already use. It becomes wrong when #3497 and #3498 flatten these onto
real `bats(1)`, where a failing assert should end one test and let the
rest report. Worth converting with that work rather than ahead of it.
`cozystack.mongodb-application` carries the identical missing operator
edge: `packages/apps/mongodb` renders `psmdb.percona.com/v1` while the
source depends only on `cozystack.networking` and
`cozystack.cozystack-engine`, so `cozystack.mongodb-operator` still
reaches no runnable suite and every change to it escalates to all 21.
Same defect class and same fix shape as the etcd edge, but it is a
second production install-ordering change and belongs in its own pull
request.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- The default etcd application now includes its operator and required
platform dependencies, ensuring the EtcdCluster provider is available.
- **Bug Fixes**
- Improved end-to-end test selection for application-specific changes,
shared components, inert paths, and unresolved or unclassified files.
- Full-suite escalations now provide clear reasons through standard
error.
- **Tests**
- Expanded coverage for test selection, dependency reachability, and
etcd application dependency declarations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
1592fd9e7c |
fix(ci): resolve fork PRs by open-PR head SHA, not commit association (#3922)
Fork PR e2e lane never ran. `resolve` looks up PR for head sha with
`listPullRequestsAssociatedWithCommit` on base repo, and this endpoint
returns nothing for fork PR head, because fork commit reaches base repo
only as `refs/pull/<n>/head` and association index doesn't walk
`refs/pull/*`. So every fork PR failed closed on `expected exactly one
open PR for <sha>, found 0`, got red required `E2E Tests` that its
author cant clear by pushing, and `publish`, `e2e`, `report` skipped.
Endpoint works fine for same-repo heads, that is why nobody saw it -
same-repo PRs get `E2E Tests` from `pull-requests.yaml` and never enter
this workflow at all. Of 935 runs of `E2E (fork)` 930 skipped as
designed, and all 5 that reached `resolve` stopped right there (three on
`found 0`, one on genuinely failed fork build). One of those reds was
later overwritten by mirrored same-repo run, so symptom got hidden
second time.
Checked on live api: on base repo that endpoint returns 0 for head sha
of all 5 currently open fork PRs and 1 for same-repo PRs, on fork repo
it returns 1 for same commit. Listing open PRs of base repo and matching
head sha returns 1.
### Fix
```js
const openPrs = await github.paginate(
github.rest.pulls.list,
{ owner, repo, state: 'open', per_page: 100 },
);
const open = openPrs.filter(p => p.head && p.head.sha === HEAD_SHA);
```
Same trust properties as call it replaces, this is plain base-repo api
data that fork cannot forge, and asking `state: 'open'` at source keeps
old rule that closed or merged PR never counts. Exactly-one guard and
head sha match are not changed, so stacked lower PR whose head is only
contained in upper branch still passes, and two PRs really sitting at
same head still fail closed. Pagination becomes load bearing here, repo
has 180 open PRs and fork PR that surfaced this sits on page two.
Most of diff is the comment block, rewritten so association call doesn't
come back later as an optimization.
Simulated new logic on live api, failing head resolves to exactly one
open PR and `labels` are present in list payload, so `full-e2e` override
still reads. actionlint, zizmor, `node --check` on extracted script and
pre-commit pass.
### What is not verified
`publish`, `e2e` and `report` never executed in this workflow, so first
fork PR going through the gate will exercise them for the first time. I
would watch digest push from OCI archives and `HEAD^2 == head_sha`
assert. Also there is no test harness for inline `resolve` script and
this PR doesn't add one.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved fork pull request detection by matching the exact triggering
commit.
* Prevented stacked or unrelated pull requests containing the same
commit from being incorrectly selected.
* Ensured the workflow proceeds only when exactly one matching open pull
request is found.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
bd4939ac5e |
docs(governance): describe sub-projects and third-party dependencies (#3784)
## What this PR does Brings `GOVERNANCE.md` in line with the organisation it describes: what the project governs, and what it ships but does not. ### Sub-projects The `Code Repositories` list named seven repositories and was wrong in both directions. Five of the six sub-projects whose code ships in the distribution were missing from it — `etcd-operator`, `keycloak-kms-proxy`, `local-ccm`, `cozystack-scheduler` and `ingress-nginx-with-protobuf-exporter` — while `talos-bootstrap` and `talos-meta-tool` were listed although an adopter installs neither. Nothing stated what being on the list meant, which is why it drifted. It is replaced by a section organised around the distinction that decides scope: whether a repository's output reaches the artifact an adopter installs. Packaged sub-projects get a table naming what each one contributes, its stability, and its owners — the same handles each repository's `CODEOWNERS` already carries. Independent tooling, project infrastructure and upstream forks are described in a line each, as explicitly outside that scope. Three things an evaluator finds anyway are stated rather than left to be discovered: that one fork (`apimachinery`) is compiled into the product, why, and that the fix is proposed upstream as kubernetes/kubernetes#135537; that four repositories are private, with the reason for each, and that none of them contributes to the distribution; and that telemetry is received by a component adopters do not install, with a link to the opt-out. ### Third-party Dependencies Writes down which shipped components the project does not govern. Cozystack vendors software it does not own, which is ordinary and needs no defence — but the boundary was nowhere stated. This is a Required item on the Incubation checklist ("All project metadata and resources are vendor-neutral"), and an evaluator currently has nothing in the governance document to read it off. The section says two things: what the arrangement is, and where the components are listed. It names nothing itself. The [Licenses](https://cozystack.io/docs/v1.6/operations/configuration/licenses/) page carries the list, and by its own framing carries upstreams only — Cozystack-maintained components are Apache-2.0 and are not listed there individually. Appearing on that page is therefore already the classification, so a second list here would only drift from it. Two components that belong on that page were missing from it despite shipping since v1.3.4 and v1.4.3; cozystack/website#649 adds them. Nothing moves, nothing is relicensed, no mirroring changes. This documents the existing arrangement, which each package's `values.yaml` already describes at its own level. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. It touches `GOVERNANCE.md` only, and no trigger concerns governance documents — the website triggers are package additions, platform values, variants, components, release assets, `ApplicationDefinition` semantics, the Talos pin and developer tooling. The website PR linked above is a companion, not a consequence of this diff: it fixes a pre-existing gap on the Licenses page that stands on its own. - [x] No downstream repository is affected by this change ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added governance guidance for project sub-components, including ownership, licensing, maintenance, and repository classifications. * Documented project infrastructure, upstream forks, private repositories, and telemetry services. * Clarified how third-party software is sourced, governed, mirrored, and versioned. * Linked to a license inventory listing third-party components and their applicable licenses. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
adc9c568fd |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs (#3887)
## What this PR does Break the tenant `bootstrap-token-tenant` Job out of its `helm.sh/hook: post-install,post-upgrade` lifecycle and move it into the main install phase, so the `bootstrap-token-<id>` Secret lands in the tenant `kube-system` before `MachineDeployment` can converge. ### Why it deadlocks today - Job wired as `post-install,post-upgrade` fires only after Helm's wait phase resolves - Wait phase blocks on `MachineDeployment` Ready - MD Ready blocks on nodes registering - Node registration blocks on the kubelet CSR path, which needs the very `bootstrap-token-<id>` Secret this Job produces - Result: any fresh install, or any HR reconcile that finds MD in a non-Ready state, sits in a deadlock; `HR.status.upgradeFailures` climbs without the post-upgrade hook ever getting its turn `pre-install,pre-upgrade` is not a fix either — those fire before non-hook templates apply, so `<release>-admin-kubeconfig` (materialised by Kamaji after `KamajiControlPlane` is applied) is not yet available. ### The shape that works Same shape `talos-reconcile` already uses in this chart: main-phase resources. Helm applies the SA and Job together with `KamajiControlPlane` and `talos-secrets`. Job's Pod stays pending until Kamaji publishes the admin kubeconfig Secret (`backoffLimit` covers the retry window). Job completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state — it doesn't gate it on MD. ### Verification - Reproduced the deadlock on two independent long-running tenant clusters (v1.6.x), documented in #3875 - Reproduced the manual unblock (creating the `bootstrap-token-<id>` Secret by hand in the tenant `kube-system`) on both — nodes register within seconds, MD converges within minutes, HR clears - Extended `packages/apps/kubernetes/tests/talos_templates_test.yaml` to pin the absence of `helm.sh/hook` on both rendered documents; `helm unittest` passes 9/9 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md`. The diff touches `packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml` and its test — no app add/remove, no `values.schema.json`, no core/platform values, no `hack/`, no CRD, no `ApplicationDefinition`, no telemetry metric. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): tenant `bootstrap-token-<id>` Secret is now created in the main install phase instead of a post-install/post-upgrade hook, unblocking fresh installs and any HR reconcile that finds MachineDeployment in a non-Ready state. ``` Closes #3875 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved bootstrap-token job scheduling by running it as a standard chart resource instead of a Helm lifecycle hook. * Jobs are automatically recreated when relevant configuration changes, ensuring updates are applied reliably. * Added startup handling to wait for the admin kubeconfig when needed. * Improved retry and cleanup behavior so deployments are not unnecessarily blocked. * Ensured related resources are created consistently during chart deployment. * Improved job naming stability while respecting Kubernetes naming limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6db7260b2a |
feat(kubernetes)!: remove FluxCD addon (#3379)
## What this PR does The Kubernetes app's optional `addons.fluxcd` toggle deployed a Flux Operator + FluxInstance into tenant clusters, served by two vendored Helm charts (`packages/system/fluxcd-operator`, `packages/system/fluxcd`). Those charts are **AGPL-3.0** — the last AGPL content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not depend on them: it runs classic Flux via `internal/fluxinstall`, and migration 21 already removed the Flux Operator from it. The addon defaulted to `enabled: false`. Rather than maintain an AGPL payload for an unused feature, this PR removes it entirely as a breaking change: - Delete both system packages and their `platform` source entries. - Remove the `addons.fluxcd` HelmReleases, values, Go types and generated schema/README from the Kubernetes app. - Add **migration 54**: for any tenant that had the addon enabled, the FluxCD HelmReleases in the management cluster are **suspended, stripped of their Flux finalizer and deleted**, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running inside the tenant keeps running untouched (mirrors the orphan idiom of migration 29). It runs as a `pre-upgrade` hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. After the change the only `affero`/`AGPL` matches under `packages/` are comments in `packages/system/monitoring` explaining why Grafana is not rebuilt — no AGPL payload ships. **Verification:** the Kubernetes app helm-unittests (181) and platform tests (88) pass; `make generate` was run with the CI-pinned `cozyvalues-gen v1.6.0` (schema, README, Go types, CRD); `migrations-target-check` and the `cozystack-version-stamp` bats pass; the `api/apps/v1alpha1` module builds. ### Downstream repositories This removes a user-facing addon and drops the `addons.fluxcd` field from the Kubernetes app's `values.schema.json`. Boxes are left unticked pending follow-ups (not opened from this PR): - **terraform-provider-cozystack** — models the Kubernetes app schema; the removed `addons.fluxcd` field should be checked/dropped there. - **website** — any hand-written mention of the FluxCD addon in the tenant Kubernetes docs should be removed (the managed-app reference regenerates from the README). ### Release note ```release-note feat(kubernetes)!: remove the FluxCD addon (`addons.fluxcd`). It relied on AGPL-3.0 vendored charts and was disabled by default; tenant clusters that had enabled it keep their running Flux release via an automatic upgrade migration. To remove that orphaned Flux from such a tenant, uninstall its `flux-operator`/`flux-instance` Helm releases inside the tenant cluster by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Breaking Changes** * Removed FluxCD addon support from tenant cluster configuration, including the Kubernetes tenant CRD schema, chart values schema, defaults, and published “Cluster Addons” parameters. * FluxCD-related Helm manifests are no longer rendered as part of the Kubernetes package. * FluxCD configuration is no longer accepted for new or updated tenant clusters. * **Migration** * During upgrade, existing FluxCD Helm releases are cleaned up automatically by suspending them, clearing finalizers, and deleting the management-side releases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
851a86dcf1 |
feat(platform): orphan tenant FluxCD HelmReleases on upgrade
Add migration 54 for the removed addons.fluxcd feature. For every tenant that had it enabled, the FluxCD HelmReleases in the management cluster are suspended, stripped of their Flux finalizer and deleted, so the delete does not trigger a Helm uninstall inside the tenant cluster. The Flux release already running in the tenant keeps running untouched. Runs as a pre-upgrade hook, before the addon-less Kubernetes chart reaches the tenant HelmReleases. Mirrors the orphan idiom of migration 29. Bumps migrations.targetVersion to 55. Fail closed. Dropping the finalizer is the only thing that keeps the delete inert, and migrations never re-run, so the script runs under `set -euo pipefail` with no `|| true` on the critical path and gates the delete on the finalizer actually being gone. The verify read uses `--ignore-not-found` rather than `2>/dev/null || true`, so a failed read aborts instead of reading as an empty finalizer list that passes the gate on an object whose state was never checked. A migration that stops is recoverable; one that half-orphans a fleet is not. Cover it with a docker + fake-kubectl unit test running the real script by path inside the migrations image's own alpine base (busybox ash, jq baked on), so it exercises the shipped interpreter and load-bearing pipefail. The suite pins ordering and selection (only the addon HelmReleases are touched, each suspend -> finalizer-drop -> delete in order) and the fail-closed paths (a surviving finalizer, a failing verify read, and a failing fleet scan each abort before deleting and before stamping). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
7cca3d75e7 |
feat(kubernetes)!: remove the FluxCD tenant addon
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a Flux Operator + FluxInstance into a tenant cluster. It was served by two vendored charts, packages/system/fluxcd-operator and packages/system/ fluxcd, which were the last AGPL-3.0 content in this Apache-2.0 repository and a governance blocker for CNCF Incubation. The management cluster does not use these charts (it runs classic Flux via internal/fluxinstall; migration 21 already removed the operator from it), and the addon defaulted to disabled. Rather than maintain an AGPL payload for an unused feature, drop it entirely: - delete both system packages and their platform source entries - remove the addons.fluxcd HelmReleases, values and generated schema from the Kubernetes app - drop the addons.fluxcd block still submitted by the e2e harness - drop the removed fluxcd package entries from CODEOWNERS BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is removed. Tenant clusters that had it enabled keep their running Flux release; see the companion migration for the upgrade path. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
4385990869 |
feat(redis-operator): migrate from archived spotahome to freshworks-oss fork (#3406)
## What this PR does `spotahome/redis-operator` was archived on 2026-06-11 (read-only; last release v1.2.4, 2022). This migrates the operator to the actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5, Apache-2.0). ### Why freshworks-oss It keeps the same `RedisFailover` CRD and API group (`databases.spotahome.com`) — so existing managed Redis is unaffected — and adds a native `spec.engine: Redis|Valkey` switch, letting managed Valkey drop its Redis-compatibility shim image and run the official `valkey/valkey` image directly. ### Changes - Re-vendor the chart from `oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which carries the `engine`-aware CRD. - Build the operator image from freshworks-oss v3.3.5 source. - Re-target the label/annotation-preservation patch onto the freshworks Go module (`github.com/freshworks/redis-operator`). ### Verification - freshworks-oss source + our `labels.diff` patch compiles natively with Go 1.26.3 (`service` and `cmd` packages). - `git apply --check` of the re-targeted patch against v3.3.5 succeeds. - The operator chart renders with the cozystack-registry image override. - The image is built (freshworks-oss v3.3.5 source + our patch, multi-arch amd64/arm64), pushed to `ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in `values.yaml`. - `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats` pass with the pinned digest. ### Note for maintainers `packages/system/redis-operator` is not part of the automated `make build`; its image is built out-of-band and the digest is pinned in `values.yaml`. That image is already built and pinned in this PR; re-run `make -C packages/system/redis-operator image` only if you bump the operator version. ### Follow-up The cozystack/redis-operator fork carries cert-manager TLS work (cozystack/cozystack#2729) that freshworks-oss does not have yet; the plan is to upstream it to freshworks-oss so it lands in the shared maintained fork rather than a cozystack-only one. ### Release note ```release-note feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for injecting additional environment variables into the Redis Operator (including a default operator group id). * Added default configurable labels for the Redis Operator. * **Bug Fixes** * Preserves and merges existing Service labels and annotations during Redis Operator updates. * **Chores** * Updated the Redis Operator and Helm chart to the Freshworks-maintained 3.3.5 release, aligned chart/image sources, and simplified Helm chart retrieval. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
7ac2ffe504 |
fix(ci): resolve fork PRs by open-PR head SHA, not commit association
The fork-PR e2e lane never ran. `resolve` looked the PR up with
`listPullRequestsAssociatedWithCommit` against the base repo, and that
endpoint returns nothing for a fork PR head: the commit reaches
cozystack/cozystack only as `refs/pull/<n>/head`, and the association
index does not walk `refs/pull/*`. So every fork PR fail-closed on
"expected exactly one open PR for <sha>, found 0", stamped a red
required "E2E Tests" its author could not clear by pushing, and skipped
publish/e2e/report.
It resolves same-repo heads correctly, which is why the hole stayed
invisible — same-repo PRs get "E2E Tests" from pull-requests.yaml and
never enter this workflow. Of 935 runs, 930 skipped as designed and all
5 that reached `resolve` stopped there; three on `found 0`, and one of
those reds was later overwritten by a mirrored same-repo run, which hid
it again.
Enumerate the base repo's open pull requests instead and match on the
head SHA. Same trust properties as the call it replaces — plain
base-repo API data the fork cannot forge — and `state: 'open'` at the
source keeps the rule that a closed or merged PR never counts. The
exactly-one guard and the head_sha match are unchanged, so a stacked
lower PR still passes and two PRs genuinely at the same head still fail
closed. Pagination stops being a corner case and becomes load-bearing:
the repo has 180 open PRs, and the fork PR that surfaced this sits on
page two.
Verified against the live API for the failing case: PR #1982 at
|
||
|
IvanHunters
|
fd8d378b73 |
fix(kubernetes): rotate bootstrap-token Job on either token half; review nits
Fold bootstrapTokenSecret into the Job content hash alongside bootstrapTokenId. talos-secrets.yaml regenerates the two token halves independently, so hashing only the id left a rotated secret no-oping a completed Job and the tenant stuck on the stale bootstrap-token. Also address review feedback: drop a review-round reference from the tracked test, correct a stale hash in a test comment, pin the alternate-image hash case exactly instead of by regex, and fix a comment indent in talos-reconcile-job.yaml. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
035bde1920 |
fix(kubernetes): correct bootstrap-token Job rationale and make it churn-free
Address review of the main-phase bootstrap-token Job. The deadlock the header claimed cannot occur on this release: the kubernetes cozyrd carries helm-install-disable-wait, which cozystack-api maps to Install.DisableWait and Upgrade.DisableWait, so there is no main-resource wait phase in front of the hooks. Rewrite the rationale around the real reason to leave the hook shape: a post-upgrade hook never runs on a release whose upgrade keeps failing, so the token Secret never lands. Fix the stale 'stays a post-install hook' note in talos-reconcile-job.yaml and the 'fails the chart upgrade loudly' comment, which is untrue once Helm no longer waits on the Job. Give the immutable Job a content-hash name suffix over its whole rendered spec, mirroring talos-reconcile: a static name would fail an upgrade that lands while the Job is live (or inside its TTL) with 'spec.template: field is immutable'. Fold the looked-up bootstrapTokenId into the hash so a rotated token also rolls the Job instead of Helm no-oping a completed one. Make the admin-kubeconfig volume optional and gate the Pod on the shared wait-for-kubeconfig init container, so it waits in init (uncounted by backoffLimit) instead of FailedMount at t0. Extend the template tests for the hashed name, its rotation on an image change, and the optional volume + init container. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
542ab899c2 |
fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs
The bootstrap-token-tenant Job was wired as a Helm post-install/post-upgrade hook. post-* hooks fire only after Helm's wait phase resolves, and the tenant kubernetes release wait blocks on MachineDeployment Ready, which blocks on nodes registering, which needs the kubelet CSR path — for which the very bootstrap-token this Job produces has to already exist in the tenant kube-system. Fresh installs and any HR reconcile that finds MD in a non-Ready state deadlock; HR.upgradeFailures climbs while post-upgrade hooks never get their turn to run. pre-install/pre-upgrade would not help either: they fire before non-hook templates apply, so <release>-admin-kubeconfig (materialised by Kamaji after KamajiControlPlane is applied) is not yet available. Move the ServiceAccount and Job to the main install phase, matching what talos-reconcile already does in this chart. Helm applies them together with the KamajiControlPlane and talos-secrets; the Job Pod stays pending until Kamaji publishes the admin kubeconfig Secret (backoffLimit covers the retry window) and then completes before MachineDeployment can converge. Helm's wait treats the Job as another resource to reach terminal state and does not gate it on MD. Extend the existing template test to pin the absence of helm.sh/hook on both the ServiceAccount and the Job, so this cannot regress silently. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
68d83cb686 |
test(e2e): add a runner-layer fixed-work canary around the node-join wait (#3919)
## What this PR does The diagnostics around the node-join wait already carry counter pairs that report time in the CPU rows, events in the KVM exits, and instantaneous values and running maxima in the files beside those. None of them is a fixed quantity of work, so none can separate a machine that was given fewer turns from a machine that took its turns and got less done with them. This adds a collector that runs two fixed amounts of work on the runner VM and times them: an integer loop and a streaming write, each under a ceiling of its own. It takes one sample before the node-join wait and one after it, on the failing and the passing path alike. Each sample is an absolute reading rather than half a difference, so one alone is still readable against the expected ranges the capture prints, and the pair says whether a slowdown began inside the interval it brackets or was already there going into it. The collector sits outside the three counter pairs that bracket the same wait rather than among them, because it occupies a core while it runs, and a burn placed inside one of those intervals would be charged to the join window it is supposed to measure. A new bats suite of 57 tests covers the collector. The existing node-join guards were extended to price it against the phase budget and to keep the spend order documented in both kubernetes suites in step with the order the block actually runs. On the budget: the sample taken before the wait is bounded at two arms of the canary ceiling plus grace, so it can add up to fifty seconds ahead of the wait, and the guard prices it at exactly that, while the guard's total stays below an honest worst case because collectors it already priced can spend more than the figures it prices them at. This branch was reviewed extensively before it was opened, and what is known to be incomplete is written down rather than left implied: the words describing how each floor sits against its healthy band are marked in the code as written rather than derived, so they are the part that can go stale when a constant moves, and the phase budget residuals are tracked in the issue the comments beside them cite. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. The diff is five files: the e2e diagnostics collector library, its two new and extended bats suites, and the two chainsaw suite definitions that document the spend order. The closest call is `cozystack/ccp`, whose skills gate on `hack/package.mk` and `hack/common-envs.mk` and drive `make generate`. Neither file is touched, nothing under `hack/` is moved or renamed, and no make target changes what it does: the new suite is picked up because `BATS_UNIT_FILES` already globs `hack/*.bats`, so the file list grows and the target's contract does not. Nothing else matches. No package is added, renamed or removed, so the website app lists are unaffected. No `values.schema.json`, version enum, default or `ApplicationDefinition` changes, so the Terraform provider is unaffected. `hack/e2e-prepare-cluster.bats` is untouched, so the node prerequisites that `ansible-cozystack` and `talm` restate are unchanged. `hack/update-crd.sh`, the `release.labels` convention and the chart source kinds are untouched, so `external-apps-example` and `cozyhr` are unaffected. No label, annotation or metric name changes, so `cozy-proxy` and `cozystack-telemetry-server` are unaffected. Minimum node and network requirements are unchanged, so `examples` is unaffected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): the node-join diagnostics now time two fixed amounts of work on the runner before and after the wait, so a run that spent its wall clock without completing work is visible in the report. No user-facing change. ``` |
||
|
Aleksei Sviridkin
|
7ec5dff93a |
test(e2e): add a runner-layer fixed-work canary around the node-join wait
The counters the node-join failure path collects are read in whatever unit their source publishes: time in the CPU rows, events in the KVM exits, instantaneous values and running maxima in the files beside those. Not one of them carries a unit of work of its own, so a machine that spent every tick and got a fraction of the work done reads healthy in all of them. Add a collector that carries its own unit of work. It runs two fixed amounts of work at the runner layer and times each: a loop over a handful of integer-valued scalars, which pressure on the shared cache or the memory controller moves little, and a store stream through blocks larger than the last-level cache one core can allocate into, which is dominated by it. Fixed work gives each reading a scale of its own, so a red run can be read with no green one beside it, and the capture states the expected range and where the estimate comes from. Sampled where the three counter pairs already are, before the wait on every run and after it on both paths, but outside their brackets: it occupies a core rather than reading a file, and a burn inside one of those intervals would land in the interval each pair divides by: runner-kernel CPU time charged to the join window, and for the two counting the guests, a core taken away from what they count. Two arms, each under a twenty-second ceiling with the five-second kill grace behind it, so fifty seconds at worst against the 420s diagnostics budget. The budget guard derives that from the ceiling the source declares, and the phase-warning guard names the canary among the collectors that keep running when timeout is missing. An exit shaped like the ceiling is attributed to it only when the whole bound was on the clock: an outside kill produces the same status, and read by status alone it would publish exactly the finding this collector exists to detect. A reading below the range the legend states, and an arm the ceiling stopped before it could produce one, reach the job log from all three reporting sites, each naming where it was taken, the way a missing reading does from the two outside the diagnostics block. On the passing path the report that holds the figure is the artifact nobody downloads. Each floor is printed in the legend it comes from and sits where a bounded arm crosses it before the ceiling stops the arm, because a floor the ceiling reaches first never fires: the arm is cut off before its rate can fall that far, and every alert then arrives as a bound rather than as the slowdown the floor was meant to name. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
899529b386 |
chore(ci): add area/cilium label and map the cilium PR scope to it (#3914)
## What this PR does Adds an `area/cilium` label and routes the `cilium` PR scope to it. Cilium has more than 20 open issues of its own right now (the pinned version and its bumps, the Gateway API implementation, network policy, IPAM next to kube-ovn, LB-IPAM), and `area/networking` no longer narrows a search. #3913 is the tracking issue that uses the label. The label already exists in the repository, created by hand so the open issues could be labelled before this lands. The sync workflow runs with `delete-other-labels: false`, so nothing is at risk in between; this PR makes `labels.yml` the owner of it and updates the scope table in `docs/agents/contributing.md`. The `cilium` scope moves from `area/networking` to `area/cilium`, following the `kamaji` and `keycloak` precedent where the dedicated area replaces the broad one rather than stacking on it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Added a dedicated `area/cilium` label for Cilium-related issues and pull requests. * Updated automated labeling and contribution guidance so Cilium changes are categorized separately from general networking work. * Refined the networking label description to exclude Cilium-related items. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
467e431765 |
chore(ci): add area/cilium label and map the cilium PR scope to it
Cilium has enough open issues of its own (version pins and bumps, the Gateway API implementation, network policy, IPAM next to kube-ovn, LB-IPAM) that area/networking no longer narrows a search. Declare area/cilium in labels.yml, route the cilium scope of PR titles to it instead of area/networking, and update the scope table in the contributing guide. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Andrei Kvapil
|
0feca20e26 |
docs(governance): describe the sub-projects and what each contributes
The Code Repositories list named seven repositories and was wrong in both directions: five of the six sub-projects whose code ships in the distribution were missing from it, while talos-bootstrap and talos-meta-tool were listed although an adopter installs neither. Nothing said what being on the list meant, which is why it drifted. It is replaced by a section organised around the distinction that actually matters — whether a repository's output reaches the artifact an adopter installs. Packaged sub-projects get a table naming what each contributes, its stability and its owners; independent tooling, project infrastructure and upstream forks are described in a line each as out of that scope. Three things an evaluator will find anyway are stated rather than left to be discovered: that one fork (apimachinery) is compiled into the product and why, that four repositories are private and none of them ships, and that telemetry is received by a component adopters do not install. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
910a917115 |
docs(governance): cut the dependency section to what it has to say
The second paragraph argued that an upstream maintained by an individual is still an upstream. Nobody had disputed it, and a governance document that defends an unraised objection invites the question instead of settling it. What remains is the arrangement itself and where the components are listed. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
95e84fc185 |
docs(governance): keep the dependency list in one place
The section named two of the upstreams it describes, which put a list here and another on the Licenses page, with nothing keeping the two in step — the drift the link was meant to avoid. States the rule and the test for applying it; the components it covers stay enumerated where they already are. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
a2f7b7956c |
docs(governance): link the licenses page from the dependency section
The section states the rule; the licenses page is where the components it covers are actually enumerated, so point at it rather than restating a list that would drift. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
ed1624cfa3 |
docs(governance): state which shipped components the project does not govern
Cozystack vendors software it does not own, which is ordinary, but the boundary was nowhere written down. Two of those components are maintained by someone who is also a Cozystack maintainer — from outside, that is indistinguishable from project code parked in a personal account, and a reader has to guess. Records the rule and names those two cases, with what makes them upstreams rather than components: their own identity and licence, a problem that exists outside this project, and absence from a default installation. Assisted-By: Claude Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
myasnikovdaniil
|
79287cd9b1 |
docs(release): add the v1.5.4 changelog, and fix v1.6.0's fail-open audit note (#3910)
Adds the changelog for `v1.5.4`, and corrects one still-live instruction in v1.6.0's. The generated changelog is a good inventory and every one of its bullets is kept. What it had no layer for is the operator: no required-actions section, no runnable pre-upgrade checks, and the SeaweedFS 4.31 rename left as three isolated fix bullets that never say what to do about it. That is what is added on top, in the shape v1.6.0's changelog uses. Base widened to `v1.5.2`. v1.5.3 was tagged but its release was never published and its changelog PR was closed unmerged, so `docs/changelogs/v1.5.3.md` exists on no ref and a v1.5.3 base leaves #3212 (persistent EFI/TPM state) and #3194 (filer postgres2 connection pool) documented nowhere at all. Coverage is 82/82 commits in range. Three claims corrected. The slot 45 divergence is a skip rather than an ordering problem: v1.5.4 is the first 1.5.x release stamped `targetVersion: 46`, and `run-migrations.sh` loops `seq CURRENT (TARGET - 1)`, so a cluster that reaches 46 runs slots 46 through 53 on the way to 1.6 and never executes 1.6's own slot 45. Its chart-side half was missing too and is not redundant, because a fresh v1.5.4 install is stamped 46 having never run any slot, so only the chart can reach that population. And the S3 checksum bullet named a barman-cloud plugin path that does not exist on `release-1.5`. The audit re-run warning names v1.6.0 rather than an earlier release, because the script does not exist at v1.5.2 at all. v1.5.4 is the first 1.5.x release to carry it, and it carries the fail-closed version. The second commit edits `docs/changelogs/v1.6.0.md`, which is deliberate rather than an accident of scope. Published changelogs are historical records and are normally left alone, but v1.6.0's upgrade notes still tell operators to run `hack/seaweedfs-naming-audit.sh`, and the copy that shipped in v1.6.0 silences every `kubectl` call with `2>/dev/null`. A timeout or an RBAC denial therefore prints an empty all-clean table indistinguishable from a genuinely clean fleet, and that output gates a runbook step that deletes PVCs. Verified per tag: v1.6.0 carries 11 blanket redirections and no `run_kubectl` wrapper, while v1.6.1, v1.6.2 and v1.5.4 carry 4 and 18, the fail-closed shape from #3436 and its backport #3474. |
||
|
Myasnikov Daniil
|
12e892325a |
docs(release): warn that v1.6.0's naming audit fails open
v1.6.0's own upgrade notes tell operators to run hack/seaweedfs-naming-audit.sh, and the copy that shipped in v1.6.0 silences every kubectl call with 2>/dev/null. A timeout or an RBAC denial therefore prints an empty "all clean" table indistinguishable from a genuinely clean fleet, and that output gates a runbook step that deletes PVCs. Anyone following those notes today can still get a false clean. Verified per tag: v1.6.0 has 11 blanket redirections and no run_kubectl wrapper; v1.6.1, v1.6.2 and v1.5.4 have 4 and 18 respectively, the fail-closed shape from #3436 and its backport #3474. Published changelogs are historical records and are not normally edited. This one is an exception on purpose, because the instruction it carries is still live and still wrong. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
eb8c19d4a1 |
docs(release): enrich the v1.5.4 changelog
The generated changelog is a good inventory but carries no operator-facing layer: no required-actions section, no runnable pre-upgrade checks, and the SeaweedFS 4.31 rename left as three isolated fix bullets that never say what to do about it. Every generated bullet is preserved; this adds to them. Widen the base to v1.5.2. v1.5.3 was tagged but its release was never published and its changelog PR was closed unmerged, so docs/changelogs/ v1.5.3.md exists on no ref and a v1.5.3 base leaves #3212 (persistent EFI/TPM state) and #3194 (filer postgres2 connection pool) documented nowhere at all. Coverage is now 82/82 commits in range. Correct three claims. The slot 45 divergence is a skip rather than an ordering problem: v1.5.4 is the first 1.5.x stamped targetVersion 46, and run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster at 46 runs 46..53 on the way to 1.6 and never executes 1.6's own slot 45. Its chart-side half was missing too, and it is not redundant -- a fresh v1.5.4 install is stamped 46 having never run any slot, so only the chart can reach it. The S3 checksum bullet named a barman-cloud plugin path that does not exist on this branch. The audit re-run warning names v1.6.0, not earlier releases. The script does not exist at v1.5.2; v1.5.4 is the first 1.5.x to carry it and carries the fail-closed version. v1.6.0 shipped the copy that silenced every kubectl failure, so a clean table from it is not evidence. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Aleksei Sviridkin
|
0b8837e552 |
fix(telemetry): make the documented opt-out cover both collectors (#3791)
## What this PR does Telemetry is reported by two components — `cozystack-operator` (cluster facts) and `cozystack-controller` (application counts) — but only the operator had a reachable switch. The documented opt-out sets `cozystackOperator.disableTelemetry` on the `cozy-installer` chart. That chart deploys the operator; it cannot reach a package the platform deploys. `packages/system/cozystack-controller` has its own independent `disableTelemetry` key that nothing propagated to, so after an operator had followed the published procedure the controller kept posting `cozy_application_count` every 15 minutes. Nothing failed, and nothing indicated the opt-out was partial. Two changes: - `packages/core/platform` gains a `telemetry.disabled` key and threads it into the `cozystack-controller` component of the `cozystack.cozystack-engine` Package — the same mechanism the bundle already uses for cilium, multus and linstor values. - `packages/core/installer/values.yaml` declares `cozystackOperator.disableTelemetry`. It was only ever referenced from the template, so the `--set` target the documentation names did not appear in the chart's own values. Both keys default to `false`. Nobody who has not opted out sees a behaviour change. The opt-out remains two steps, because the two collectors are installed by two different charts. Collapsing it into one knob would mean the operator writing the value into the platform Package it does not own, which is a design change rather than a fix. `helm unittest` on the platform chart: 137 passed (133 before, plus the four added here). `helm lint` clean on both charts. ### Screenshots Not a UI change. ### Downstream repositories Trigger map walked against the diff. `packages/core/platform/values.yaml` changed, which the map routes to the website's hand-written platform-package table; the same change also makes the published opt-out procedure wrong, so the docs follow-up covers both. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/651 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(telemetry): the documented telemetry opt-out now covers both collectors. Setting `cozystackOperator.disableTelemetry=true` alone left `cozystack-controller` reporting application counts; the platform gains a `telemetry.disabled` key that silences it. Both default to false. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform telemetry opt-out setting, disabled by default. * Added a separate operator telemetry opt-out setting, also disabled by default. * Applying the platform setting now disables telemetry reporting for the relevant controller. * **Bug Fixes** * Ensured telemetry settings are correctly applied across supported platform variants while preserving existing configuration overrides. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
4b50a3ceb4 |
fix(ingress-nginx): pull the protobuf-exporter from the project namespace (#3783)
## What this PR does
Points the `protobuf-exporter` sidecar at the project's registry.
The sidecar was still pulled from `ghcr.io/kvaps/...` while the
controller beside it — built from the same repository, pinned to the
same `v1.11.2` — already came from `ghcr.io/cozystack/...`. The
repository moved to the organisation some time ago and the image
reference did not follow. It is the last thing in the tree that ships a
Cozystack-owned component out of an individual's namespace, which is a
Required item on the Incubation checklist ("All project metadata and
resources are vendor-neutral").
The image is the same bytes: `v1.11.2` was copied into the organisation
namespace with the manifest digest preserved, so `sha256:6d9235a9…` is
unchanged here. Nothing is rebuilt and nothing is upgraded.
The version is deliberately not bumped. `v1.11.5` exists in the
organisation namespace, but the tag tracks the upstream ingress-nginx
version and the controller sits on `v1.11.2`; bumping only the sidecar
would split the pair.
The comment in `hack/promote-retag.sh` loses its `ghcr.io/kvaps/...`
entry, because that path now falls under the bullet below it — a
cozystack-org repository outside `$REGISTRY`
(`ghcr.io/cozystack/cozystack`). Selector behaviour is unchanged;
`hack/promote-retag_test.bats` passes, 11/11.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The two files touched are a package `values.yaml` image reference
and a comment in `hack/promote-retag.sh` — no trigger matches. In
particular the `ccp` trigger is a move, a rename, or a change to what a
make target does; this is neither, and its anchors (`hack/package.mk`,
`hack/common-envs.mk`) are untouched.
- [x] No downstream repository is affected by this change
### Release note
```release-note
fix(ingress-nginx): the protobuf-exporter sidecar is now pulled from ghcr.io/cozystack instead of a personal namespace. Same image and digest; no version change.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated the protobuf exporter image source to the new container
registry location.
* Refreshed image promotion guidance to remove an outdated third-party
registry example.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
cozystack-ci[bot]
|
c6238e1544 |
docs: add changelog for v1.5.4
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Andrei Kvapil
|
3197c48bc5 |
test(migrations): cover migration 54 redis adopt (label+pin, CRD probe fail-closed)
Drive migration 54 end-to-end against a fake kubectl in a jq-enabled build of
the migrations image's alpine base (busybox ash, the production interpreter),
mirroring the docker+fake-kubectl migration-test harness.
Pinned properties:
- an absent CRD stamps 55 without patching anything;
- a failing CRD probe aborts under set -euo pipefail before stamping (a
one-off apiserver blip must not read as "absent" and skip adoption);
- every RedisFailover across namespaces is labelled onto the freshworks
operator-group, while the engine is pinned to redis:6.2.6-alpine only where
the image is empty (both roles for Harbor; sentinel only where redis.image
is preset, which is left untouched; label-only when both are preset);
- a failed adoption patch aborts before stamping so the Job retries.
Auto-discovered by the Makefile hack/*.bats wildcard; no registration needed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
|
||
|
Andrei Kvapil
|
c9ee4f6de8 |
test(platform): pin redis-operator upgradeCRDs=CreateReplace
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
f2bb0db1fe |
docs(redis): point at the freshworks-oss operator fork
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
c7a3320bb4 |
fix(migrations): pin the engine and fail-close the CRD probe in migration 54
Two defects in migration 54's adoption pass: (1) The label landed before the image pins, so the operator adopted a labelled CR with empty image fields and rolled Redis to the fork's 7.2.4 default — rewriting dump.rdb into a format the later 6.2.6 chart pin can no longer load. Pin spec.redis.image/spec.sentinel.image to redis:6.2.6-alpine in the same pass, only where empty. (2) The CRD probe used a bare `get crd >/dev/null 2>&1` that could not tell NotFound from an API blip and stamped anyway; switch to migration 50's `--ignore-not-found -o name` so a real error aborts under set -e. Also self-contained the renumber note and qualified the resource. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
caf9511365 |
fix(platform): CreateReplace the redis-operator CRDs on upgrade
helm-controller skips CRD upgrades by default, so an upgraded cluster would keep the old spotahome schema and prune the fork's spec.engine field on admission. The schema diff is additive, so CreateReplace is safe. Also refresh the harbor e2e wait comment for the freshworks operator. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
3e250ebfde |
test(harbor): assert the operator-group label and pinned engine
Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
fde96dd7da |
fix(redis): pin the Redis engine at 6.2.6 in both RedisFailover charts
The freshworks-oss operator's compiled-in default image moved from redis:6.2.6-alpine to 7.2.4; Harbor pinned no image at all and apps/redis left its sentinels unpinned, so an upgrade would move them silently. 7.2.4 rewrites dump.rdb into a format 6.2.6 refuses to load, so a platform rollback would leave Harbor's Redis crash-looping on its own data. Pin redis+sentinel (Harbor) and the sentinel (apps/redis) at the version they run today; bump deliberately and separately. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
0a3a3bfa77 |
fix(migrations): adopt existing RedisFailovers onto the operator group
The freshworks-oss operator watches RedisFailovers through a server-side operator-group label selector, so a CR without the label is silently dropped from its informer. The chart templates now stamp the label, but an existing CR only gets it when its own HelmRelease next upgrades — after the operator rolls — leaving a window with no healing, and a suspended/stuck HelmRelease never gets it at all. Label every RedisFailover in a pre-upgrade migration, ahead of the new operator. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
023134a20d |
fix(harbor): label the Redis failover with the operator group
Harbor provisions its own RedisFailover; the freshworks-oss operator filters by the redis-failover.freshworks.com/operator-group label, so without it Harbor's Redis is never created and harbor-core CrashLoops. Stamp the same group (cozystack) as the operator and the redis app. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
77cb19c8db |
fix(redis): label RedisFailover with the operator group
The freshworks-oss operator filters RedisFailovers by the redis-failover.freshworks.com/operator-group label matching its OPERATOR_GROUP_ID; without the label it ignores the CR and no Redis is provisioned. Stamp the group (cozystack) so the migrated operator adopts the app's clusters. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
0b57261d38 |
fix(redis-operator): set required OPERATOR_GROUP_ID for the freshworks fork
The freshworks-oss fork exits fatal at startup when operator_group_id is unset, unlike the archived spotahome operator. Pin a stable non-empty value via extraEnvVars so the operator boots instead of CrashLooping and blocking the install. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
abb38ab72f |
chore(redis-operator): pin built freshworks v3.3.5 image digest
Build the freshworks-oss v3.3.5 operator image (with our label-preservation patch) and pin its digest in values.yaml, replacing the placeholder tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
fe01ee17bc |
feat(redis-operator): migrate from archived spotahome to freshworks-oss fork
spotahome/redis-operator was archived 2026-06-11 (read-only). Switch the operator to the actively-maintained freshworks-oss/redis-operator fork, which keeps the same RedisFailover CRD and API group (databases.spotahome.com) and adds a native Valkey engine (spec.engine: Redis|Valkey) — so managed Redis is unaffected and managed Valkey no longer needs a Redis-compat shim image. - re-vendor the chart from oci://ghcr.io/freshworks-oss/charts/redis-operator - build the operator image from freshworks-oss v3.3.5 source - re-target the label/annotation-preservation patch onto the freshworks module The operator image is built out-of-band (make -C packages/system/redis-operator image) and its digest pinned in values.yaml; a maintainer must rebuild and pin before merge (tag is a placeholder here). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
myasnikovdaniil
|
f4ab9ec8da |
docs(release): add changelog for v1.6.2 (#3907)
This PR adds the changelog for release `v1.6.2`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.6.2.md`.
|
||
|
cozystack-ci[bot]
|
be6e66786f |
docs: add changelog for v1.6.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
Aleksei Sviridkin
|
7194edf625 |
fix(kubernetes): validate MachineHealthCheck condition timeouts at render time (#3901)
## What this PR does Follow-up to #3898, closing two gaps in the render-time duration guard it introduced. Context: #3513. First, the guard now also covers the MachineHealthCheck condition timeouts (`readyUnknownTimeout`, `readyFalseTimeout`) in both the kubernetes and kubernetes-nodes charts. They need it more than the autoscaler annotation does: the CAPI webhook validates `nodeStartupTimeout` with a 30s floor but does not validate `unhealthyConditions` timeouts at all, so a zero or negative value used to apply silently and remediate a Machine the moment its Ready condition flipped, which for a KubeVirt worker means deleting the VM. Second, the guard accepts the canonical form a Go duration serializes back to ("20m0s", "1h0m0s"). `metav1.Duration` prints that shape, so a value copied from `kubectl get -o yaml` output previously failed the render. The first segment must still be positive and later segments may be zero, so every accepted string still parses to a positive duration and the overflow and fraction-to-zero rejections stay. Mechanically, the guard moves into a `kubernetes.positiveDuration` named template in each chart's `_helpers.tpl`, and the error message format is unchanged. Tests cover the canonical form on all three fields and the rejection of unitless, zero, and negative condition timeouts in both charts. The previous test case that passed unitless values through verbatim is reworked into rejections: a unitless value used to fail only at CAPI decode on apply, and a zero or negative one did not fail anywhere. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): reject zero and negative MachineHealthCheck condition timeouts at render time, and accept the canonical Go duration form ("20m0s", "1h0m0s") for the worker join timer values ``` |
||
|
Aleksei Sviridkin
|
3710541278 |
fix(kubernetes): validate MachineHealthCheck condition timeouts at render time
The CAPI webhook validates nodeStartupTimeout with a 30s floor but does
not validate unhealthyConditions timeouts at all, so a zero or negative
readyUnknownTimeout or readyFalseTimeout applied and remediated a
Machine the moment its Ready condition flipped. Route both fields
through the same render-time duration guard as maxNodeProvisionTime,
extracted into a kubernetes.positiveDuration helper in each chart.
The guard now also accepts the canonical form a duration serializes
back to ("20m0s", "1h0m0s"): metav1.Duration prints that shape, so a
value copied from kubectl output previously failed the render. The
first segment must still be positive and later segments may be zero,
which keeps every accepted string a positive duration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
|
||
|
Aleksei Sviridkin
|
b32ca5cbb3 |
chore(kubernetes): drop the force-delete flag that blocks partial cleanup (#3899)
## What this PR does Removes `--force-delete-unregistered-nodes=true` from the tenant cluster-autoscaler Deployment. On the clusterapi provider the flag cannot do what its name promises: the provider's `ForceDeleteNodes` returns `cloudprovider.ErrNotImplemented`, the autoscaler falls back to the ordinary `DeleteNodes` path, and that path enforces the node group's minimum size. The one behavior the flag does change is harmful here: it skips the step that trims a long-unregistered delete batch down to the group's headroom above min size, so a batch larger than the headroom is refused whole where the default would trim it and delete what fits. Context: #3513. A regression test renders the Deployment, proves the args list is live with a positive anchor on `--cloud-provider=clusterapi`, and asserts the flag is absent, so reintroducing it fails the suite. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note chore(kubernetes): drop the cluster-autoscaler force-delete-unregistered-nodes flag; the clusterapi provider does not implement force deletion, and the flag only made batch removal of long-unregistered machines all-or-nothing ``` |
||
|
Aleksei Sviridkin
|
245bca3366 |
chore(kubernetes): drop the force-delete flag that blocks partial cleanup
`--force-delete-unregistered-nodes` reads as an override that lets the autoscaler remove a long-unregistered machine below its node group's min size. On the clusterapi provider it grants nothing of the sort: force deletion is not implemented there, the call falls back to the ordinary delete path, and that path refuses to take a group below its min size. The flag never bought the safety valve its name suggests. What it does buy is a behaviour nobody asked for. With the flag set the autoscaler skips trimming the delete batch down to the group's headroom above min size, and the ordinary delete path then refuses a batch larger than that headroom outright instead of removing what it can. A group of three machines above a min size of one loses two long-unregistered machines with the flag unset, and none with it set. Removing it restores the documented default and takes away a signal that advertised a protection the provider does not offer. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
5fe9450145 |
fix(kubernetes): widen and expose the worker node join timer budgets (#3898)
## What this PR does Worker node join on nested tenants regularly outruns the timer budgets the charts shipped, and the timers then destroy capacity that was about to arrive: the cluster-autoscaler treats a machine as long-unregistered after its built-in 15m default and orders a replacement, and the MachineHealthCheck deletes a Machine that is still legitimately booting. The timer audit in #3513 (https://github.com/cozystack/cozystack/issues/3513#issuecomment-5329896688) walked every timer on the join path against the cozyreport artifact of a red run; this PR sizes the budgets from those measurements and makes the hardcoded ones configurable. Context: #3513. Three changes, one commit each: - Every worker MachineDeployment in both the kubernetes and kubernetes-nodes charts now carries the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, rendered from a new `maxNodeProvisionTime` value defaulting to "30m". The autoscaler starts this clock when the machine enters its node-instances cache with a real providerID, not at Machine creation, and on a value it cannot parse it silently falls back to its 15m default rather than failing, so the template validates the value at render time and rejects anything that does not parse to a positive duration. - The MachineHealthCheck `unhealthyConditions` timeouts become values (`readyUnknownTimeout` default "5m", `readyFalseTimeout` default "10m"; nested under `nodeHealthCheck` in the kubernetes chart, flat in kubernetes-nodes) instead of the hardcoded 30s and 300s. 30 seconds of `Ready=Unknown` deletes a worker whose guest was merely starved of CPU, and 300 seconds of `Ready=False` is less than a CNI rollout takes on a loaded host. - The `nodeStartupTimeout` default rises from "10m" to "20m". The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets up to 20m for the disk image import and the VM to come up, and then up to 20m again for the guest boot and kubelet registration. The `maxUnhealthy` docs now also spell out the round-down trap in small groups: at "50%" a group of one never remediates. The budgets are coherent with the measured phases: in the audited runs a healthy disk import took 2m22s and import plus VM start 2m53s, while the stuck runs spent over 17 minutes holding a running VM with no Node, so 20m per MachineHealthCheck leg and 30m on the autoscaler clock absorb that shape of slowness without letting a genuinely dead machine linger much longer. New helm-unittest suites cover the annotation rendering, the render-time duration guard (including the values the guard exists for: unitless numbers, unknown units, zero, negatives, parser overflow), and the MachineHealthCheck timeout rendering in both charts. The render-parity test no longer pins the MachineHealthCheck values on either side and instead compares the two charts' defaults, so a future drift between parent and child defaults fails the suite. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: TODO-LEAD-FILE-ISSUE (suggested title: "Kubernetes/KubernetesNodes: model the new readyUnknownTimeout, readyFalseTimeout and maxNodeProvisionTime fields (upstream cozystack#NNNN)"; the provider models node_health_check for both resources and gains no typed access to the three new fields until it learns them) - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): widen the worker node join timer budgets and make them configurable: the cluster-autoscaler maxNodeProvisionTime is now set per MachineDeployment (default 30m), the MachineHealthCheck Ready=Unknown and Ready=False timeouts are values (defaults 5m and 10m), and nodeStartupTimeout defaults to 20m ``` |
||
|
Aleksei Sviridkin
|
ddb15ebc6a |
fix(kubernetes): widen the worker node startup budget to 20m
nodeStartupTimeout is the window a Machine has to produce a Node at all, measured from the later of the Machine's creation and its InfrastructureReady transition. What has to fit inside it is the guest boot and the kubelet registering with the tenant apiserver, and on a host under load neither is quick. At 10m a worker that was still booting was deleted before it ever joined, and its replacement started the same race from the beginning. Also record what a percentage maxUnhealthy does at small group sizes, because the interaction is not what the field name suggests: the value is scaled against the machines the check covers and rounded down, and remediation proceeds only while the unhealthy count is at or below the result. At the default "50%" a group of one never remediates, and a group of two remediates one unhealthy machine but not two. The golden-parity gate pinned the child chart's MachineHealthCheck values to the parent's defaults as inputs, which made a default moved on both sides fail there for the wrong reason. Drop the pins so the comparison is between the two charts' defaults, which is the contract it exists to hold, and pin the defaults themselves in each chart's own unit tests instead. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
Aleksei Sviridkin
|
33701800d6 |
fix(kubernetes): make the worker MachineHealthCheck timeouts configurable
The two unhealthyConditions timeouts were fixed in the template at 30s for `Ready=Unknown` and 300s for `Ready=False`, and both are too tight for a worker joining on a loaded host. `Ready=Unknown` is what the node lifecycle controller writes when a kubelet stops reporting, on a grace period of its own, so 30s on top of that deleted the VM of a guest that had merely stopped being scheduled for a stretch, before it had a chance to answer again. And `Ready=False` is the condition a registered Node holds until its CNI is up, so 300s was the entire budget a first boot got for the CNI rollout to reach it -- a worker deleted mid-rollout takes the rollout with it and its replacement starts over. Move both to values, defaulting to 5m and 10m. The remediation of a worker that is genuinely dead is correspondingly later, which costs less than it appears to: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on either of these timers. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |