mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
7777
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
0d4a3e3134 |
refactor(backups): make MongoDB PITR option uniform with CNPG + warn on unknown restore options
PITR: replace the bespoke spec.options.pitr{type,date} blob with a single
spec.options.recoveryTime RFC3339 string, matching the CNPG strategy's option
exactly (CNPGRestoreOptions.RecoveryTime). The driver converts recoveryTime to
the psmdb Restore's pitr{type:date,date} in the operator's "YYYY-MM-DD HH:MM:SS"
UTC format. An empty recoveryTime restores the snapshot as-taken (documented
difference from CNPG, whose empty value means latest WAL — a psmdb logical
backup is already consistent). A non-RFC3339 recoveryTime fails terminally.
Unknown options: parseMongoDBRestoreOptions now also returns the set of
spec.options keys the driver does not recognise; reconcileMongoDBRestore emits a
UnknownRestoreOption Warning event per unknown key (and logs it) instead of
silently ignoring it — so a "recoverytime" typo no longer silently degrades a
PITR restore to a snapshot restore. (BackupJob carries no spec.options blob, so
there is no backup-path equivalent; the only backup-time input is the free-form
BackupClass parameters used as template vars.)
Docs + examples updated to recoveryTime; tests cover the RFC3339→UTC mapping,
empty/invalid recoveryTime, and unknown-key reporting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
|
540bb94f58 |
test(backups): cover createMongoDBBackupArtifact producer seam
Address branch-review blocker: add TestCreateMongoDBBackupArtifact_ArtifactShape, mirroring the MariaDB sibling. createMongoDBBackupArtifact writes the driver-metadata keys (backup-name/namespace/destination), the Status.Artifact URI, and the persisted snapshot that the restore path reads back — the producer half of a seam previously exercised only end-to-end. A key-name drift between this producer and resolveMongoDBBackupSource would otherwise pass every unit test and surface only in e2e. The test asserts the metadata keys, phase, artifact URI, and that the snapshot decodes to a usable by-reference backupSource; a second case pins the empty-destination path (nil Artifact, no destination key). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
bc5e47c64f |
fix(backups): add explicit MongoDB cleanup-on-delete branch + its guard test
Address branch-review recommendation: give the MongoDB strategy an explicit case in BackupReconciler.cleanupOnDelete, matching CNPG / Altinity / MariaDB / FoundationDB. A deleted MongoDB Backup artifact previously fell through to the Velero default (a safe no-op — MongoDB driverMetadata never carries velero.io/backup-name), but the explicit branch documents the "we do not own the psmdb S3 archive" contract and guards the seam against a future shared helper incidentally stamping velero.io/backup-name onto MongoDB metadata. Ships with TestBackupCleanup_MongoDB_DoesNotFallThroughToVelero, mirroring the one-test-per-branch convention its three siblings follow: it seeds the velero metadata keys + a matching velero.io/Backup and asserts no DeleteBackupRequest is synthesised and the Velero Backup survives — so deleting the branch makes the test fail, which is what makes the guard self-verifying. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
dff56aa998 |
feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so `apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and ClickHouse native drivers — logical backups, restore-to-differently-named instances, and point-in-time recovery through the unified BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545). The driver orchestrates the operator's PerconaServerMongoDBBackup / PerconaServerMongoDBRestore CRDs: - BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls status.state, and persists the S3 destination + storage descriptor (by-reference, no credentials) onto the Cozystack Backup artifact. - RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back from the source backup status, snapshot fallback) so the same artifact restores in-place or into a differently-named instance; PITR via spec.options.pitr (date/latest). Because psmdb resolves the backup storage by name against the cluster's spec.backup.storages, the driver names the chart's storage (s3-storage) via the cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear Ready=False precondition when a target has backups disabled. Wiring: - api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy. - internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp partial-schema mirrors; dispatch, supported-kinds and scheme registration. - backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb Strategy CR, and psmdb.percona.com RBAC. - apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no CA-bundle field) so backups can target Cozystack's self-signed seaweedfs. Tests + e2e: - unit tests for the driver (validation, ensure-idempotency, precondition, backupSource resolution, snapshot round-trip, PITR option parsing). - examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that drives run-all.sh end-to-end (sentinel document round-trips through S3 into a restored copy while the source stays untouched), per docs/agents/e2e-testing.md. - docs/operations/backup-classes.md updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
a242f34593 |
fix(clickhouse): assign DB_LIST so backup retention cleanup runs (#3760)
## What this PR does The legacy per-instance ClickHouse backup CronJob (rendered when `backup.enabled=true`, `backup.schedule` is set and `backup.useSystemBucket=false`) never applied its retention policy. `packages/apps/clickhouse/templates/backup-script.yaml` printed `echo DB_LIST=$(...)` instead of assigning `DB_LIST`, so the variable stayed unset and the cleanup loop `for db in $DB_LIST` ran zero iterations — `restic forget` and `restic prune` were never invoked, snapshots accumulated in object storage indefinitely despite the configured `cleanupStrategy`, and the Job still exited 0 so nothing surfaced the failure. This drops the stray `echo` so `DB_LIST` is assigned from the table list and the retention cleanup runs as configured. MariaDB's equivalent backup script already assigns `DB_LIST` correctly; this brings ClickHouse in line. A helm-unittest regression over the rendered `backup.sh` asserts the assignment is present and the `echo DB_LIST=` typo is gone. Verified with `make -C packages/apps/clickhouse test` (18/18 pass). Fixes #3758. ### Screenshots <!-- No UI changes. --> ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The change is a one-line shell fix inside a Helm template plus a helm-unittest case — it touches no `values.yaml`, `values.schema.json`, `README.md`, `ApplicationDefinition`, version enum, variant/bundle, release asset, `hack/` layout or namespace. No downstream repository restates any of this. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(clickhouse): ClickHouse backup retention now prunes old snapshots. The scheduled backup job's cleanup step never ran because `DB_LIST` was left unset, so snapshots accumulated in object storage indefinitely; it now honors the configured retention (`cleanupStrategy` / `--keep-last=N`). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Fixed backup processing so table lists are assigned correctly, allowing retention cleanup to run as expected. * Removed unintended output of the shuffled table list during backups. * **Tests** * Added regression coverage to verify correct backup list assignment and prevent the previous malformed assignment from recurring. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
2b5c85cde7 |
fix(clickhouse): assign DB_LIST so backup retention cleanup runs
The legacy per-instance backup script printed `echo DB_LIST=$(...)` instead of assigning DB_LIST, leaving it unset. The cleanup loop `for db in $DB_LIST` then ran zero iterations, so `restic forget` and `restic prune` never executed and snapshots accumulated in object storage indefinitely despite the configured cleanupStrategy. The Job still reported success, hiding the failure. Assign DB_LIST from the shuffled table list so the retention cleanup loop runs. Add a helm-unittest regression asserting the rendered backup.sh assigns DB_LIST and no longer contains the `echo DB_LIST=` typo. Fixes #3758 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d04231829a |
docs(hack): fix a wrong claim about grep-error propagation in two contract tests (#3620)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does `hack/promote-gate-contract.bats` and `hack/release-freeze-contract.bats` both carried the same wrong comment above their `code_lines` helper: it said a real grep error (exit 2) fails a pin through the helper's own exit status, distinct from an empty match (exit 1). That was never actually the mechanism. `code_lines` is always piped into a further `grep` in both files, never the last stage of a pipeline, so ordinary (no `pipefail`) pipe semantics discard whatever it returns on its own, regardless of runner. `hack/cozytest.sh`, the runner these tests actually execute under in CI, also appends a `return 0` to every line that is exactly `}`, `code_lines`' closing brace included, making that exit status even less trustworthy as a signal. What catches a real grep error is the output, and only for one class of pin. The error empties `code_lines`' stream, so a pin that requires something to be present in that stream fails on it. A pin demanding an absence stays blind: an exact count of zero is what an emptied stream produces anyway, and what the pin was written to assert. Both files hold pins of that shape. Nor is the `[ -n "$block" ]` guard at the top of each test the backstop: a block here is a bare awk extraction, upstream of `code_lines`. The new comment says both things and stops there. Comment-only: `code_lines`'s body is untouched in both files, and all tests pass under both `bats` and `hack/cozytest.sh` before and after. Known limitation of the wording: the comment says the top-of-test `[ -n "$block" ]` guard cannot see a filter failure, which is true, but both files also carry non-empty guards on values that did pass through `code_lines` (`promote-gate-contract.bats:72,73,165` and `release-freeze-contract.bats:166,236,327` via `step_line`), and those do fail on an emptied stream. The discriminator is whether the guarded value came through the filter, not where the guard sits in the test. ### Screenshots Not applicable, no UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified comments describing how code-line filtering errors are detected. * Documented pipeline status behavior and why output-based assertions are used instead of relying solely on command status codes. * No user-facing behavior or executable logic changed. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
cf6bf2e379 |
fix(ci): stop labeled events from cancelling and re-running backports (#3569)
## What this PR does A run joins its concurrency group before any job-level `if` is evaluated, so `prepare`'s guard in `backport.yaml` is too late to help. By the time it works out that a label event is irrelevant, that run has already cancelled whatever was in flight for the same PR. Labels set with the default `GITHUB_TOKEN` start no run, which is why the in-repo labelers never showed this. Third-party GitHub Apps do start runs, and they arrive in bursts. The delivery survived, which is why this went unnoticed for so long. The old guard admitted a run on the PR's cumulative label set, so the last label of a burst, the one nothing cancels, requalified and redid the work. In the bursts on record the kill lands within seconds of the merge run starting, well before the cherry-pick, so the cost was a restart rather than a lost backport, recorded as nothing more than a cancelled run sitting next to a green one. Two changes to the concurrency key, and both are needed. A label event that requests no backport now goes into a group of its own, so it can neither cancel nor displace the run doing the work. A label event that does request one queues behind that run instead of killing it, because `backport` and `backport-previous` are separate requests and both have to finish. Splitting alone would let `backport` cancel the merge-triggered run. Queuing alone would let an unrelated label evict a pending request, since the group holds a single pending run and a newer one replaces it. `prepare`'s condition is scoped in the same commit. It read the PR's cumulative label set, so once a merged PR carried `backport`, any later unrelated label re-entered the job to redo a backport already delivered. That run used to get cancelled by the next label in the burst. After the change above it runs to completion, so the scoping has to land here rather than as a follow-up. The coupling runs the other way too, and it is the sharper half. That redundant run is exactly what redelivered the backport the burst had just killed, so narrowing the guard on its own would convert a noisy delivery into a missing one. Neither change is safe to land without the other. Both jobs also gain a `timeout-minutes` ceiling, which is a consequence of the queuing rather than housekeeping. A cancelling key disposed of a stuck run by killing it; queuing makes the next genuine request wait behind it instead, on the six-hour job default, which turns one wedged job into a six-hour hole in the release line. That ceiling reaches a stuck job and nothing else: a run can also wedge at the run level with every job already finished, and `timeout-minutes` has no run-level equivalent, so nothing in the workflow reaches that state. `docs/release.md` now tells an operator to look for a run still in flight and clear it before re-applying the label, because the retry queues behind it rather than replacing it. The ceiling bounds execution, not the wait for a runner, and thirty minutes is generous rather than calculated. Execution on run 29844315422 was six seconds for `prepare` and two and sixteen for the matrix legs, against 9m24s, 5m28s and 5m45s of runner queue for the same three jobs. That queue is a third unbounded case: GitHub's limits page says "job execution time" for the six-hour job cap while spelling out that the 35-day run cap "includes execution duration, and time spent on waiting and approval", and [orgs/community#50926](https://github.com/orgs/community/discussions/50926) shows a job sitting six hours on "Waiting for a runner to pick up this job" under `timeout-minutes: 5`. So the value is picked to leave room for a cherry-pick far larger than any on record while replacing the six-hour default, and nothing here is sized against the queue, because nothing here can be. This removes a mitigation, and the trade is worth naming on both sides. Under an unconditional cancel a run wedged at the run level cleared itself, because the next event killed whatever was sitting in the group. That is the same behaviour that killed live backports, so the mitigation is inseparable from the bug being fixed: keeping it means keeping label events that cancel a backport in flight. What it buys is a change of failure mode rather than one fewer failure. A backport lost silently, discovered weeks later as a fix missing from a release line, becomes a delay the operator can see in the run list, with the recovery written into `docs/release.md`. There is no lossless alternative here: `queue: max` removes the eviction but only by dropping the conditional cancel entirely, and the group key cannot tell a manual retry from a second genuine request. The run cited above, 31232645133, is still live and retained on purpose: it is the only observed instance of the run-level wedge, so clearing it would remove the evidence for the paragraph that describes it. Its exposure is one pull request, #3262, which carries no backport label, so nothing is queued behind it. The old condition also carried a disjunct that could never decide anything. A `labeled` payload already lists the new label in `pull_request.labels`, so `contains` was true whenever the explicit `github.event.label.name` check was, and the name check only ever agreed with a `contains` that had already matched. The restructure drops it by construction rather than by deletion: the name check is now the whole of what a `labeled` event may match on. ### Why this no longer touches `pull-requests.yaml` `main` fixed it there by splitting the group key, then refined that so a `full-e2e` label stays in the main group instead of publishing a second `E2E Tests` status on one SHA. That is the better mechanism, it neither queues nor delays, and `gate-concurrency-contract.bats` now pins `cancel-in-progress: true` in that file, so what this PR originally did would turn it red. Nothing was left to add there, so the hunk is dropped and its pin with it. The split-key idea is applied in `backport.yaml` instead, where it did not exist. One claim from the earlier description is withdrawn. It said the leftover risk was a duplicate backport PR, opened when the original had merged and its branch had been deleted before the queued run pushed. `main` has since gained a guard that lists closed PRs on the `backport-<n>-to-<target>` branch and skips the target when one of them merged. Branch deletion does not defeat it: `GET /branches/backport-3510-to-release-1.6` is 404 while the same head filter on `/pulls?state=closed` still returns the merged PR. So a redundant run only cost runner minutes, and the scoping above removes the run. ### Screenshots Not applicable, no UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The only workflow it cites is `tags.yaml`, for the release-prep behaviour `cozystack/ccp` documents. Every `hack/` trigger it lists either names a specific file another repository mirrors or gates on, or covers moving and renaming under `hack/`. This PR appends tests to a contract file the map does not name, and moves nothing. No package, chart, values file, CRD, image reference, namespace, variant or node prerequisite in the diff. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(ci): stop `labeled` pull request events from cancelling the automatic backport run already in flight. A burst of labels from a third-party app could previously kill the backport the merge had just started, leaving the work to be redone by whichever label arrived last. ``` |
||
|
|
4ac02f59d5 |
fix(e2e): stop the sandbox cluster CIDR overlapping the pod range (#3750)
## What this PR does The sandbox Talos config and the platform install step both used `10.244.0.0/16`, one as the Kubernetes cluster CIDR and the other as the kube-ovn `POD_CIDR`, so two allocators drew from one range. kube-ovn owns pod IPAM and allocates flat from `POD_CIDR`. Cilium is chained behind it (`generic-veth`, target `kube-ovn`) and allocates no pod addresses at all, but it still carves its own per-node router IP and Ingress IP out of `Node.spec.podCIDR`, which kube-controller-manager slices from the cluster CIDR. Sooner or later kube-ovn hands a pod an address cilium already holds. When that address is the Ingress IP the agent refuses the sandbox with `IP ipv4:<addr> is already in use`, and cilium never relents on it, so the pod proceeds only if something reschedules it onto a different address; in the run this was filed from, the same address was refused seven times across six pod UIDs before the suite ran out of budget. When it is the router IP nothing complains and only host-to-pod traffic breaks. This moves the sandbox cluster CIDR to `100.65.0.0/16`, which no other range in the tree uses, and adds a guard so the two cannot be brought back together by accident. Pod addresses do not change. They still come from kube-ovn out of `10.244.0.0/16`, and cilium does not require a chained endpoint's address to sit inside its own allocation CIDR, so only cilium's two per-node infrastructure addresses move. The quiet variant gets fixed twice over, which is worth spelling out because only half of it was obvious. Under `ipam.mode: kubernetes` cilium also installs a route for the node's own pod CIDR via `cilium_host`. On the current default that route covers `10.244.a.0/24` on a node whose pods actually live behind `ovn0`, so it shadows real pod addresses. Moving the cluster CIDR both stops the router IP from ever being handed to a pod and retargets that route at `100.65.a.0/24`, where it shadows nothing. Excluding the addresses on the kube-ovn side instead is not an option: cilium picks its offsets by random scan over the node's range, so there is no static set to exclude. The collision shows up in two shapes, and both have been seen. Loudly, on the Ingress IP, where the agent names the address in the event, which is the failure https://github.com/cozystack/cozystack/issues/3736 was filed for. Quietly, on the router IP, where the pod comes up and the only symptom is `connection refused` on a kubelet probe, with `ip route get <podIP>` on the node resolving to `local ... dev lo`; that variant is documented in https://github.com/cozystack/cozystack/issues/3445 and https://github.com/cozystack/cozystack/issues/3615. They were being tracked as unrelated flakes in different suites before the shared cause was identified. ## What the guard covers, and what it does not `hack/sandbox-cidr-disjoint.bats` pins the configuration, not the behaviour. It reads the values out of the tree and fails when they overlap. It does not reproduce the collision, which needs a running cluster. It checks the invariant rather than the literals, so either value may move as long as they stay disjoint, and it checks the cluster CIDR against the service and join ranges too, because landing on one of those instead is the next way to get this wrong. An extraction that comes up empty, duplicated or misshapen fails the test instead of shrinking the comparison, and comments and blank lines inside the list are skipped rather than read as the end of it. It is not a YAML parser, though: anything it does not read as a bare dotted quad fails the shape validation rather than being understood, which is the safe direction but not the same promise. Those three are the allocator ranges and nothing more. The sandbox node network, the load-balancer pool and the transit range of a component that is off by default are not read, so the guard is not a complete map of what the cluster CIDR has to avoid. An IPv6 entry fails the shape validation rather than getting an overlap check, so dual-stack means extending the guard rather than passing through it. ## Residual risk I did not prove from source that cilium is the only consumer of `Node.spec.podCIDR` in this topology. What I did check is that kube-proxy is disabled and that `local-ccm` runs the node-lifecycle controller only, with no route controller. This diff selects the full e2e suite, 21 Chainsaw suites with `gateway` and `mongodb` among them, so a consumer I missed surfaces as a red run on this PR rather than staying an assumption. Split of what covers what: the static checks and the unit guard cover the configuration, and the e2e run covers the behaviour. Nothing short of a green suite answers whether cilium's router and Ingress addresses behave once they sit outside the range kube-ovn allocates from. ## A check that is not "the suite went green" `hack/e2e-cilium-endpoint-leak-healer.sh` already prints a line of its own for this exact collision, so there is a signal to read that does not depend on the overall verdict. In the `cozyreport` artifact it lands at `snapshots/<suite>/host/namespaces/kube-system/v1/pod/cilium-leak-healer-*/healer/current.log`, and the lines to grep for are `IP is this node's Cilium ingress IP`. In the run this was filed from there are 42 of them, all naming the same address. After this change that line cannot be produced by an address inside the kube-ovn range, because there is no longer an address that is in both. So the expectation is zero from the first run, not zero on average: one occurrence contradicts the change rather than being noise, and is worth stopping on. A green suite can be green for many reasons; this line disappearing has only one. ## Scope This fixes the sandbox and nothing else. For a real installation the cluster CIDR lives in the operator's own machine configuration, outside this repository, so the same collision is still reachable there. https://github.com/cozystack/cozystack/issues/3740 tracks the product side, including the finding that the install documentation does not merely allow the overlap but instructs operators to create it, by requiring `podCIDR` to match the cluster's pod CIDR. There is a tradeoff in this worth naming rather than leaving to be discovered. Talos defaults `podSubnets` to `10.244.0.0/16` (`DefaultIPv4PodNet`, `pkg/machinery/constants/constants.go:888` at v1.13.6, the version this tree ships), and the platform defaults `networking.podCIDR` to the same value, so an operator who takes both defaults gets the overlap. Until now the sandbox shared that topology and was the one place the bug surfaced where anyone would see it. After this change it no longer reproduces the operator-facing form, and nothing else does either. That is the right call for CI, which should not be wedged by a defect it did not introduce, but it does mean the production exposure loses its only accidental detector. Closes https://github.com/cozystack/cozystack/issues/3736 To be explicit about what closing that issue does and does not mean: this removes the reproduction in the sandbox and leaves the product configuration untouched. The collision is still reachable for users, and https://github.com/cozystack/cozystack/issues/3740 is where that lives. The issue text describes the root mechanism, which is wider than its title, so the closure covers the e2e failure it was filed for and not the platform default. ### Screenshots Not a UI change. ### Downstream repositories No box is ticked deliberately, and no follow-up PR is opened, so a human decides. Walking the trigger map against this diff: both `hack/e2e-prepare-cluster.bats` entries in it are scoped to node prerequisites, the kernel modules a node must load, the containerd settings, the LVM `global_filter` and the cluster domain. This diff changes none of those, it changes a network range, so no entry fires literally. The underlying constraint does reach several of them all the same. `cozystack/talm` sets `podSubnets: [10.244.0.0/16]` in `charts/cozystack/values.yaml`, the same range the platform uses for `networking.podCIDR`, so the documented Talos bootstrap produces the overlap this PR removes from the sandbox. `cozystack/ansible-cozystack` reaches it from the other side: `roles/cozystack/defaults/main.yml` sets `cozystack_pod_cidr: "10.42.0.0/16"` and renders it into `networking.podCIDR`, while its inventories set `cluster-cidr: 10.42.0.0/16`, so those two are identical as well. And `cozystack/website` prescribes the equality outright, `install/kubernetes/generic.md` stating that `podCIDR` and `serviceCIDR` must match the Kubernetes cluster configuration. No issue is filed there yet on purpose: which side moves is not decided, and asking talm to change `podSubnets` would be the wrong request if the platform ends up moving instead. That decision belongs to https://github.com/cozystack/cozystack/issues/3740. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Updated cluster networking ranges to avoid conflicts with Kubernetes, kube-ovn, kilo, and Cilium networks. - Added validation to detect missing, invalid, or overlapping cluster and platform CIDR ranges. - Preserved safeguards for environments where network ranges overlap, with clearer diagnostic logging. - **Documentation** - Documented network separation requirements and clarified expected sandbox networking behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9ec5f8ceca |
fix(migration): carry logSerialConsole through worker-pool adoption
Migration 54's adopt pick-list copied every worker NodeGroup field that feeds the content-hashed KubevirtMachineTemplate except logSerialConsole. That field renders into the same hashed template (kubernetes-nodes nodegroup.yaml), so an adopted pool that set logSerialConsole would render a KMT without it, change the content-hash, rename the KMT, and roll every worker VM of the pool on upgrade (and silently drop the setting). Add logSerialConsole to the pick-list (guarded by has(), so pools without it are unaffected) and assert it carries through. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
e0d6c6be92 |
fix(ci): stop labeled events from cancelling and re-running backports
A run joins its concurrency group before any job-level `if` is evaluated, so `prepare`'s guard cannot stop a label event from disturbing the run already backporting a PR. Labels set with the default GITHUB_TOKEN start no run, but labels set by third-party GitHub Apps do, and they arrive in bursts, so each one cancelled the backport the merge had just started. The delivery survived anyway, which is why this went unnoticed: the old guard admitted a run on the PR's cumulative labels, so the last label of a burst requalified and redid the work. In the bursts on record the kill lands within seconds, well before the cherry-pick, so the cost was a restart rather than a lost backport. Move a label event that requests no backport into a group of its own, and let one that does request a backport queue rather than cancel. Both halves are needed. `backport` and `backport-previous` are separate requests that both have to complete, so they share a group and must not cancel each other, while every other label does no work here and only has to stay out of the way. Scope `prepare` to the labels each trigger is entitled to read at the same time. Reading the cumulative label set on a `labeled` event let any later unrelated label on a merged PR still carrying `backport` re-enter the job to redo a backport already delivered. That redo is also what redelivered a killed backport, so narrowing the guard alone would turn a noisy delivery into a missing one: neither change is safe to land without the other. Bound both jobs with `timeout-minutes` for the same reason, since a queued request now waits on the run ahead of it rather than replacing it. The ceiling bounds execution and not the wait for a runner, which no job-level key reaches, so 30 is generous rather than calculated: every execution measured on this workflow has been seconds, against minutes of queue that nothing here can bound. Document the recovery path the narrowing removes: with no accidental re-run left, re-applying whichever backport label the PR already carries is how an operator retries a merge-triggered run that died. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
ec6a835251 |
fix(manifestutil): report one message when the CRD wait is cancelled (#3742)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does `WaitForCRDsEstablished` reported a cancelled context two different ways. The check at the top of the poll loop returned a message with no CRD name, the `select` at the bottom named the CRD the poll had stopped on. When a tick and the deadline come ready in the same instant, `select` picks between the ready cases at random, so one cancellation reached the operator either way. Both callers wrap this as `CRDs not established after apply`, so the CRD name is what makes that log line actionable, and whether it is there was a coin toss. `TestWaitForCRDsEstablished_timeout` asserts on the name, which is where this surfaced: its 2s deadline is an exact multiple of the 500ms ticker, so the two channels come ready together on the fourth tick and the assertion failed at random. The message is now built in one place and returned from both exits, so the two are indistinguishable by construction rather than by inspection. The name is carried across iterations, because a cancellation noticed at the top of the loop has no poll of its own to take one from, and it starts as the first CRD of the set for a wait cancelled before any poll observed anything. That start value is a stand-in rather than an observation, and the comment on it says so; the alternative, an empty name and a second phrasing for it, would leave the loop-top message with nothing to assert against and no test able to catch its return to the unnamed form. Deleting the top check instead, so there is literally one exit, is the smaller diff and it moves the name to the head of the list in the very interleaving this PR is about. A client refuses `Get` on a dead context at the first name in the list whichever CRD is actually lagging, so a poll entered on a cancelled context renames the report to that first name. That premise is about the production client, `client.New` at `cmd/cozystack-operator/main.go:176`, uncached, so the request is rejected before it is issued, and it is worth saying that nothing in the test surface pins it: replacing the `ctx` passed to `Get` with `context.Background()` survives the whole package, and the fixtures that need a context-honouring client check `ctx.Err()` by hand. That line predates this change and is untouched by it. Measured with a client that honours the context: with the check, a wait that re-enters the loop after the deadline keeps the name its last poll settled on; without it, the report moves to the head of the list. The check closes that window and no more than that. A context that dies part-way through a poll still records the first `Get` to fail, exactly as merge base does, because this branch cannot tell a refused request from a CRD status. That is unchanged here and written up separately. Four tests cover the ways this wait can end on a cancelled context: before any poll has observed the set, cut off inside a poll, cancelled while waiting for the next tick, and re-entering the loop on a context that died during that wait. Two of them are red on merge base and are the regression pins for the defect above, both deterministically rather than by repetition: merge base answers a wait that starts cancelled, and a wait that re-enters the loop cancelled, with the unnamed message every time. Its loop-top exit is a `select` whose `Done` case is ready, and a ready case always beats `default`. The middle two are green on merge base, which gets those cases right; they are here because each catches a specific way of breaking the fix. Every one of the four is the sole killer of at least one mutation, which is the property worth having: it says none of them is redundant. They compare the whole message rather than searching it for a name, because a second exit that happens to carry a name is still the defect being fixed here, and a test that only looks for the name cannot see it. The re-entry test needs the bottom `select` to take the tick while cancellation is already pending, which against a real context is the same coin toss the bug rides on. The double it uses hands out an open `Done` for the single read that `select` makes in that iteration and a closed one for every read after, so the tick wins by construction and the next iteration sees a context cancelled through both `Done` and `Err`, the way a real one is. That pairing is guaranteed rather than incidental: `cancelCtx.Err` reads `<-c.Done()` before returning a non-nil error, so a double that reported cancellation through `Err` alone would be modelling a state no context can be in, and would pin the shape of the check rather than its behaviour. One property of that double is worth knowing before editing the loop: it hands out its open and closed channels per read, so a `Done()` read added inside the poll would consume the open one and leave the bottom `select` a closed channel, quietly turning this into a second bottom-exit test that still passes. The failure direction there is false-green, which is the worse half, and the cheap guard is a lower bound on elapsed time inside that test, a run that leaves through the loop top spends a ticker interval getting there, one that leaves at the bottom of the first iteration does not. Not done here, because the test is correct today and provably kills the mutation it exists for, so the hazard is conditional on a future edit. The timeout test does reproduce locally under load: 6 failures across 400 runs on the unfixed code while I was measuring, spread over five batches, the last of which saw none. #3719 reports that it does not reproduce locally, so this is worth writing down before someone spends an evening re-testing that. That spread is also why the number is an order of magnitude and not a rate. `TestInstall_crdNotEstablished` in `internal/crdinstall` has the same 2s deadline against the same 500ms ticker. It asserts on the `CRDs not established` wrapper the caller adds, and both messages carried that wrapper, so this coin flip could not reach it and nothing here touches it. I left the 2s deadline where it is. Moving it off the tick boundary would have hidden the coin flip and left the second message in place. The new tests catch a reintroduced second message deterministically, which is what that job now rests on rather than the timeout test's roughly one-in-sixty. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(manifestutil): a CRD wait cut short by a cancelled context now reports one error naming the CRD it was waiting on, where before the name was present or missing depending on which of two exits the scheduler happened to pick ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cancellation handling while waiting for custom resources to become established. * Cancelled operations now stop before issuing unnecessary requests and identify the affected resource. * Timeout errors remain distinguishable from cancellation errors. * **Tests** * Added coverage for cancellation before, during, after, and between polling attempts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9ace63b441 |
fix(e2e): stop the sandbox cluster CIDR overlapping the pod range
The sandbox Talos config and the platform install step both used 10.244.0.0/16, one as the Kubernetes cluster CIDR and the other as the kube-ovn POD_CIDR, so two allocators drew from one range. kube-ovn owns pod IPAM and allocates flat from POD_CIDR. Cilium is chained behind it and allocates no pod addresses, but it still carves its own per-node router IP and Ingress IP out of Node.spec.podCIDR, which kube-controller-manager slices from the cluster CIDR. Sooner or later kube-ovn hands a pod an address cilium already holds. When that address is the Ingress IP the agent refuses the sandbox with "IP ipv4:<addr> is already in use", and cilium never relents on it, so the pod only proceeds if something reschedules it onto a different address. In the failure this comes from, the same address was refused seven times across six pod UIDs before the suite ran out of budget. When it is the router IP nothing complains at all and only host-to-pod traffic breaks. Move the cluster CIDR to 100.65.0.0/16, which no other range in the tree uses. That deliberately stops the sandbox matching the layout the install documentation prescribes, so it is not a value to put back. Pod addresses are unaffected: they still come from kube-ovn out of 10.244.0.0/16, and cilium does not require a chained endpoint's address to sit inside its own allocation CIDR. Excluding the addresses on the kube-ovn side would not work, because cilium picks its offsets by random scan over the node's range. The new guard checks the invariant rather than the literals, so either value may move as long as they stay disjoint. It covers the service and join ranges too, since landing on one of those instead is the next way to get this wrong. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
8886ffa410 |
test(e2e): drop the parent-chart heredoc cases removed by the Phase 2 split
Phase 2 moved the worker talos-reconcile Job and its TalosConfigTemplate out of the kubernetes chart into kubernetes-nodes, so the cases that rendered packages/apps/kubernetes --show-only templates/talos/talos-reconcile-job.yaml now fail with 'could not find template'. Keep the kubernetes-nodes cases, which cover the same heredoc escaping (hostile registryMirrors + Talos image coordinates) against the chart that actually renders the worker Job now. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
ea89cb188a |
test(e2e): match the guest-console-log assertion to the Phase 2 KubernetesNodes CR
Since the Phase 2 split the e2e worker pool is a separate KubernetesNodes CR whose fields sit two spaces under spec, not six spaces under a nodeGroups.<name> block, so the logSerialConsole assertion is re-anchored to the two-space key. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
ec2203a934 |
docs(security): publish vulnerability reporting and a backlog reconciliation
Four independent reviews of the first draft of this reporting found that it overstated remediation, understated how much of the backlog was never applicable, and contained a disclosure leak the stated rule had not anticipated. What lands here is the corrected version. The reconciliation is the substantive artifact. It audits every finding the CVE-scanning pipeline has tracked against what the project actually shipped, because the pipeline has no write-back from this repository and so has never marked anything fixed. Findings: remediation did happen and is invisible to the tracker, concentrated in one wave in June 2026 rather than the steady stream an earlier draft claimed; 57% of the backlog was never applicable or cannot be assessed, the largest block being packages that contain no executable vulnerable code, chiefly kernel headers, since a container never provides a running kernel; reachable exposure in a stock install is an estimated 2-5% of the tracker; four component bumps are available upstream and not yet taken; and the pipeline itself has seven defects, including deduplication by CVE identifier alone and a broken attribution step that left 16% of findings unattributable. The monthly reports are cut back to per-month facts. Remediation is no longer attributed to individual months: crossing a severity-banded count of still-open findings with one month's component bumps, which anyone can enumerate from the commit log, narrows the search for unpatched exposure more than a bare count does. An earlier draft did exactly that while refusing to publish bare counts on disclosure grounds. The triage-latency figure is also gone, because 494 of 495 decisions were recorded in a single bulk pass and a per-finding latency cannot be recovered from a closing timestamp. Two process-tracking issues that the pipeline's severity filter should have excluded are no longer counted as findings, so the total is 825 rather than 827. The disclosure rule is revised in two directions, both more open: findings with no upstream fix will be named with mitigation guidance, since there is nothing to wait for and a mitigation is then the only available control; and the generator's guard now covers every advisory namespace rather than CVE alone, refuses to emit any file if any file would leak, and takes released evidence rather than a stale tracker label as its source of truth. SECURITY.md, the self-assessment and the GTR are brought into line: all three previously described this reporting as unpublishable, and the GTR refused on principle the counts the reports print. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
b4a2f6f791 |
Merge remote-tracking branch 'origin/main' into pr-3315-lgtm
# Conflicts: # api/apps/v1alpha1/kubernetes/types.go # api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go # hack/e2e-chainsaw/_lib/run-kubernetes.sh # packages/apps/kubernetes-nodes/tests/render-parity.sh # packages/apps/kubernetes/README.md # packages/apps/kubernetes/templates/cluster.yaml # packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml # packages/apps/kubernetes/tests/talos_templates_test.yaml # packages/apps/kubernetes/values.schema.json # packages/apps/kubernetes/values.yaml # packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml # packages/system/kubernetes-rd/cozyrds/kubernetes.yaml |
||
|
|
d0990299ed |
feat(clickhouse): let the backup sidecar trust a private S3 CA, unblocking the backup e2e round-trip (#3385)
## What this PR does Gives the ClickHouse `clickhouse-backup` sidecar a way to trust an S3 endpoint whose certificate is signed by a private CA, and uses that to make the ClickHouse backup/restore e2e round-trip actually run in CI. This actualizes #2600, whose feature code and example scripts have since landed on `main` independently, and whose remaining contribution — the e2e test — was still in the retired BATS format. #2600 can be closed once this merges. ### 1. `backup.endpointCA` on the ClickHouse chart The sidecar had **no TLS/CA surface at all**: bucket coordinates and credentials arrive as `S3_*` env, and nothing let it verify an endpoint signed by a private CA. Cozystack's own in-cluster SeaweedFS is exactly that case — it serves `:8333` behind the self-signed "SeaweedFS CA" — so a ClickHouse release could only ever back up to a publicly-trusted endpoint. MariaDB (`storage.s3.tls.caSecretKeyRef`) and Postgres (barman `endpointCA`) have had a CA field all along; this closes the gap for ClickHouse. `backup.endpointCA` takes `{name, key}`. When set, the chart mounts that Secret into the sidecar and points `SSL_CERT_DIR` at it. Empty — the default — renders no env, no mount and no volume, so existing releases are byte-for-byte unchanged. **Why `SSL_CERT_DIR` and not `AWS_CA_BUNDLE`.** Both are honoured by `altinity/clickhouse-backup:2.7.4`; I measured them against a self-signed endpoint and public AWS S3 in turn. Go reads `SSL_CERT_DIR` *in addition to* the system bundle, so both endpoints verify. `AWS_CA_BUNDLE` *replaces* the SDK's trust pool: the private endpoint verifies, and public S3 then fails `x509: certificate signed by unknown authority` — it would silently break a release that also talks to a public endpoint. `S3_DISABLE_CERT_VERIFICATION` was rejected outright, since it drops verification rather than trusting one more CA. ### 2. The e2e suite, and what it was really testing Two Tests added to the existing `hack/e2e-chainsaw/clickhouse/` suite (in name order, mirroring the `etcd`/`mariadb`/`postgres` shape — see the layout rationale below for why they live here rather than in a new directory): - **`clickhouse-1-backup-contracts`** (CI, `tenant-test`, ~98s): a backup-enabled ClickHouse asserted against the *running* cluster — the `clickhouse-backup` sidecar on the `ch-backup-api` `:7171` port, and the `<release>-backup-api-auth` Secret whose keys the sidecar's `API_USERNAME`/`API_PASSWORD` actually resolve `valueFrom`. Each is a silent break if a chart bump renames it. Needs no object store. - **`clickhouse-2-backup-roundtrip`** (CI, `tenant-root`): `Bucket` → `Altinity` strategy + `BackupClass` → source ClickHouse with a sentinel row → `BackupJob` → in-place `RestoreJob` → to-copy `RestoreJob` into a second instance, driving `examples/backups/clickhouse/run-all.sh` so test and docs cannot drift. The round-trip was originally gated behind `CLICKHOUSE_E2E_S3_ROUNDTRIP` and **skipped in CI** — its `PASS (1.01s)` meant the gate fired, not that a backup happened. `backup.endpointCA` removes the reason for the gate, so the suite now adopts the same three preconditions mariadb/postgres rely on: run in `tenant-root` (an isolated tenant's Cilium egress allowlist blocks its Pods from reaching `tenant-root`'s SeaweedFS), target the in-cluster endpoint instead of the `Bucket`'s external ingress URL (an unroutable placeholder in CI), and trust its certificate via the copied CA. It also pre-cleans with `cleanup.sh`, because `tenant-root` is shared and a leftover `Succeeded` `BackupJob` would satisfy the harness's wait without a backup ever running. Both backup Tests live in the existing `hack/e2e-chainsaw/clickhouse/` suite rather than a new suite directory, so no selector mapping is added: `hack/select-e2e.sh` already derives `clickhouse` both from `packages/apps/clickhouse/**` (the `*-application` rule) and from `examples/backups/clickhouse/**` (the harness path), and both paths therefore select the suite that actually runs the harness. That alignment is the reason for the layout — a separate `clickhouse-backup/` directory would have made an edit to the harness select a suite that cannot exercise it. `hack/select-e2e.sh` and `hack/select-install.sh` are otherwise untouched (the only change is a comment recording that invariant), and the existing `backup example harness edit selects its app suite` case in `hack/select-e2e_test.bats` covers the mapping. ### Verification - `helm unittest` — 17 pass, including 5 new `endpointCA` cases: the unset default renders nothing, the env/mount/volume wiring, a custom key still projected as `ca.crt`, composition with `useSystemBucket`, and `backup.enabled=false` renders nothing. - The CA-trust behaviour was measured against the real image, not assumed: without a CA knob the sidecar reports `x509: certificate signed by unknown authority`; with `SSL_CERT_DIR` the handshake succeeds. - `hack/select-e2e_test.bats` (16) and `hack/select-install_test.bats` (17) pass; `select-install.sh --validate` reports graph OK. - `make generate` is idempotent, so the pre-commit job's `git diff --exit-code` stays clean. - The contracts Test passed in CI on earlier runs of this branch. The round-trip only became CI-live with this work, so it still needs a green E2E run before merge — the branch has since been rebased onto current `main` to get one. ### Screenshots Not applicable — no UI changes. ### Downstream repositories Walked the trigger map against the diff, file by file. The load-bearing change is the new `backup.endpointCA` field in `packages/apps/clickhouse/values.yaml` / `values.schema.json`; everything else is `hack/e2e-chainsaw/`, `examples/backups/`, one docs paragraph, and regenerated artifacts, none of which appear in the map. - [ ] No downstream repository is affected by this change - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/18 — "add, remove or rename a field in an app's `values.schema.json`" reaches the provider's hand-maintained schema, model and expand/flatten pair, so the new nested object is not surfaced until it is added there. Opened as an issue rather than a PR because the natural moment is alongside that repo's next `api/apps/v1alpha1` pin bump, which is a maintainer call. Not ticked, with reasons rather than silence: **cozystack/website** — the trigger is *adding, renaming or removing a package*, and this adds a field to an existing one. `clickhouse` is already in the website `Makefile`'s app list and its reference page is generated from this package's `README.md`, so the regenerated `backup.endpointCA` rows are carried by the existing release-tag automation with no manual PR. **cozystack/external-apps-example** — vendors `hack/package.mk` and `update-crd.sh`, neither of which this PR touches. **cozystack/ansible-cozystack**, **ccp**, **talm**, **cozyhr**, **cozy-proxy**, **cozystack-telemetry-server**, **examples** — no installer values, `hack/` layout, namespace, variant, node-prerequisite, annotation, label or metric in their trigger rows is touched here. ### Release note ```release-note feat(clickhouse): add `backup.endpointCA` so the clickhouse-backup sidecar can verify an S3 endpoint whose certificate is signed by a private CA, such as Cozystack's in-cluster SeaweedFS. Leave it unset for publicly-trusted endpoints; existing releases are unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional private S3 endpoint CA support for ClickHouse backup/restore sidecars via `backup.endpointCA`, including configurable Secret name and CA bundle key (default `ca.crt`). * Sidecars mount the CA and extend trust while leaving public endpoints unchanged. * **Documentation** * Updated ClickHouse chart docs, operational backup guidance, and backup/restore examples to cover CA setup, overrides, and cleanup. * **Tests** * Added helm-unittest and end-to-end coverage validating CA rendering/mounting behavior and ClickHouse backup/restore workflows. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
e1fc2ab82e |
docs(security): fix the supported set at three lines and record that no backport rule exists
Maintainer decisions on the two questions this branch raised rather than answered. Supported versions are the latest stable minor and the two minors preceding it. Expressed as a rolling rule, so it names concrete lines when read against the Releases page without going stale, and it replaces the activity-window test, which classified every 1.x line back to 1.0 as supported and so described something wider than the project does. There is no documented rule for choosing which supported lines a fix is backported to; it is a per-fix maintainer decision weighing severity against backport risk. Say that plainly in SECURITY.md instead of implying a policy. That also resolves the promise in the incident-response process, which said a fix ships across all affected supported release lines. It does not necessarily, and with no rule governing the choice that sentence committed the project to something it has not agreed. Point at the per-fix decision instead. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
c1da673b08 |
docs(security): align the reporting paragraph with the undocumented backport rule
The vulnerability-reporting paragraph named severity and upgrade impact as the basis for backporting to an older line, two paragraphs after the table says no rule is documented. Both sentences were added by this branch, so this is an internal inconsistency of its own making rather than inherited text. Point at the same absence instead of naming a basis. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
f2a091083a |
docs(security): drop an invented backport rule and fix the nightly counts
The previous commit added a backport prioritisation policy to SECURITY.md — by severity, by reachability in a default configuration, and by adopter count, with a critical fix reaching every supported line. No such rule exists in any project document. Reachability appears only as a triage question, and adopter count appears nowhere. In an adopter-facing file that also lists seven supported lines, that sentence read as a guarantee no maintainer has made. Replaced with what is actually true: backporting is decided per fix and the rule is not documented. Nightly figures corrected. Counting only scheduled runs, the first run after the fix still failed, the streak that followed was three days rather than four, and six of the last twenty-four passed rather than seven. The earlier numbers folded in a workflow_dispatch success from a day whose scheduled run failed. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
ea3d68493a |
docs(security): restore the triage admission and correct the nightly timeline
The previous commit narrowed a true statement into a false one. It said the scanner gap was remediation throughput and that triage ran on schedule; triage is also behind, and the backlog awaiting a triage decision includes findings at critical severity well past the one-business-day target. Both tracks are now stated as behind. Figures stay out of this repository for the disclosure reason the document already gives and belong in the private channel, so the admission is qualitative rather than rounded down. The nightly characterisation was also wrong. It read as fixed-then-regressed, which understates it: after the fix in #3332 the scheduled run passed for four days, went red for five, passed for three, and has failed every run since 2026-08-02 — seven of the last twenty-four passed. It is flapping, not newly broken, and the current failure is untriaged rather than provably unfiled. Carry the target-attainment caveat into SECURITY.md, which stated the triage SLAs with no indication they are not being met, and resync the self-assessment on the monthly report: it names findings individually and so is not publishable until reduced to counts by severity. Also note in SECURITY.md what the activity test actually classifies today — every 1.x line from 1.0 to 1.6 — and how backport effort is prioritised across that set, since supporting seven lines in parallel otherwise invites the question it does not answer. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
a627788b81 |
docs(security): make the disclosure line and the support test checkable
A second internal pass found that the previous commit's disclosure argument did not hold together and that its support test could not be evaluated by a reader. The document had nominated the monthly aggregate summaries as safe-to-publish evidence on the grounds that they hold no vulnerability detail. They do: they name accepted-risk findings individually by CVE ID, severity and package, some of them critical, next to a count of confirmed findings awaiting a fix. State the line where it actually falls — aggregate counts by severity are publishable, naming an individual unfixed finding is not, because a CVE ID plus a package plus this project's public pinned component list locates unpatched exposure — and note that the monthly summary needs reducing to counts before it can be published. Present the resulting posture as a proposal rather than as a position the maintainers have taken. The scanner SLA paragraph said a triage backlog was running against the clock. The pipeline's own output says something more specific and worse: triage is keeping up, remediation is not, with the last monthly period closing on no confirmed findings fixed. Say that instead. The supported-versions table keyed support to having an active release branch. Release branches are never deleted, so every line back to the earliest has one and the test tells a reader nothing; read literally it also committed the project to every 1.x line. Key it to patch activity within a stated window instead, restore an end-of-life category the previous commit had removed entirely, and put the v0.41 cutoff back rather than extending discretionary backports across all of 0.x. Also: drop an unevidenced claim that the per-PR checks are healthy, correct the nightly attribution — the current red run is a regression after #3332 and is not tracked by an issue, unlike the older failure in #3278 — and qualify how zizmor gates, through the required pre-commit hook rather than through its own path-filtered workflow. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
d30d638575 |
docs(security): correct scanner, Renovate and support-line claims after internal review
Two internal reviews of the previous commit found that it overstated several controls. Corrections, all verified against the tree and the GitHub API: CodeQL is not a required status check on main — the required checks are pre-commit and E2E Tests — so the questionnaire no longer says it gates merges. SECURITY.md and the security self-assessment already hedged this correctly and the GTR now matches them. Renovate does not cover Helm charts: helm-values is disabled repo-wide because many image refs are assembled by the cozy-lib.image helper rather than written literally in values.yaml. Say so, and say that the curated platform component images are bumped by maintainers per release rather than by a bot. The rewritten supported-versions table narrowed the support window to the latest two minors, which contradicts practice — several 1.x lines receive patch releases concurrently. Restate it as a description of what the project actually does, keyed to active release branches rather than to version numbers. The CVE-scanning pipeline is described more precisely: noise filters run before issue creation, discovery reads direct Go requirements with transitive coverage coming from the filesystem scan, and render-time image refs are a known blind spot. Its SLAs are labelled as targets rather than measured attainment, and the confidentiality claim is narrowed from the whole repository to per-CVE findings and triage state, since the policy documents and scripts hold no vulnerability data and are what actually evidences the control. SBOM and provenance are stated from the build tooling rather than from an issue title: provenance is explicitly disabled and SBOM generation is implemented but defaulted off. zizmor is added as a third gating scanner in both documents. Also close markers that repository evidence settles: the in-tree security self-assessment, threat model and incident-response documents; no default audit policy is shipped; and there is no top-level NOTICE file, which is stated as a compliance gap rather than deferred. Fix the phantom OWNERS reference at its source in CONTRIBUTOR_LADDER.md, and add current CI health next to the e2e workflow links so the red nightly run is disclosed rather than discovered. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
be8c6e507f |
docs(security): address GTR review — rolling supported-versions policy, CODEOWNERS, SCA and CI links
Review feedback on the GTR questionnaire flagged two factual errors and a set of unresolved markers that the repository can in fact answer. The supported-versions table in SECURITY.md pinned concrete version numbers and had drifted five minor releases behind the latest release. Restate it as a rolling policy relative to the latest release so it cannot go stale, and cite the Releases page as the authoritative list instead of repeating numbers in prose. The contributor-ladder answer referenced a top-level OWNERS file, which the project does not have; promotions go through .github/CODEOWNERS for the area-scoped Reviewer role and MAINTAINERS.md for the Maintainer roster. Close the markers that repository evidence settles: e2e and CI workflow links, the Renovate configuration, the organization-wide Trivy CVE-scanning pipeline and its triage SLAs, and the real state of artifact signing, SBOM and SLSA provenance. Replace the vague item count in the draft banner with the seven substantive gaps that genuinely need measurement or a maintainer decision. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
|
24157ec876 |
chore(codeowners): keep architects as owners in every non-governance rule (#3738)
## What this PR does CODEOWNERS applies only the **last** matching rule, and matching is not additive: a scoped rule fully replaces the `*` catch-all instead of extending it. As a result `@kvaps` and `@lllamnyp` were not code owners for most scoped paths (21 and 36 of the 76 owner-bearing rules, respectively), and several paths (`/hack/`, `/.github/`, `/Makefile`, `/.pre-commit-config.yaml`, `/docs/`) were owned by exactly two people — with "Require review from Code Owners" enforced on `main`, a PR authored by one of them could only be approved by the other. This PR: - appends `@kvaps` and `@lllamnyp` to every rule that lists owners, keeping existing owners and their order; the governance block deliberately stays with `@tym83 @kvaps`, and the ownerless rules for generated files stay ownerless; - spells out the non-additive last-match semantics in the file header; - adds `hack/codeowners-invariant.bats` (auto-discovered by `make bats-unit-tests`), so a future scoped rule that omits the catch-all owners fails unit tests instead of silently carving its path out from under them. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note chore(codeowners): @kvaps and @lllamnyp are now code owners for every path except governance files, removing single-approver review bottlenecks; a unit test guards the invariant ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated code ownership rules across dashboards, packages, Kubernetes components, backups, GPU and SeaweedFS components, Keycloak, documentation, and CI files. * Expanded area-specific review ownership while preserving existing governance and API review assignments. * **Tests** * Added automated validation to ensure ownership rules include the required default reviewers. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5747747087 |
fix(manifestutil): report one message when the CRD wait is cancelled
WaitForCRDsEstablished reported a cancelled context two different ways. The check at the top of the poll loop returned a message with no CRD name; the select at the bottom named the CRD the poll had stopped on. Which one ran was left to the scheduler: when the deadline and a tick come ready in the same instant, a select picks between the ready cases at random, so one cancellation reached the operator either way. Build the message in one place and return it from both exits, so the two are indistinguishable by construction. The name is carried across iterations, since a cancellation noticed at the top of the loop has no poll of its own to take one from, and starts as the first CRD of the set for a wait cancelled before any poll observed anything. Keeping the check at the top is what makes carrying the name worthwhile. A client refuses Get on a dead context at the first name in the list whichever CRD is actually lagging, so a loop that re-entered and polled anyway would overwrite the name its last poll settled on. The check closes that window and no more than that: a context dying part-way through a poll still records the first Get to fail, because this branch cannot tell a refused request from a CRD status. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
cfaf2e086c |
fix(backupstrategy-controller): repair lookup-gated backup objects (#3524)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Fixes #3518. Objects in `packages/system/backupstrategy-controller` and `packages/system/bucket` are gated on a Helm `lookup` of an object that is still being created. When the lookup is empty the object is skipped — and since helm-controller does not re-render a release whose chart and values did not change (drift detection is off on operator-generated HelmReleases), **the skip is permanent**, not a bootstrap window. On the reporter's cluster the install-timing window was 4 seconds and the objects were still missing months later: - the 7 default backup `Strategy` CRs and the Velero `BackupStorageLocation`, gated on `include "backupstrategy-controller.bucketName"` → a `lookup` of `BucketClaim`; - the `<bucket>-<user>-credentials` Secret (`packages/system/bucket/templates/user-credentials.yaml`), gated on a `lookup` of the COSI Secret. Only `BackupClass cozy-default` was present, because it renders unconditionally. Its own comment predicted this case while assuming "Flux re-renders the chart", which does not happen — the same trap `packages/core/platform/sources/cozystack-basics.yaml` already documents: *"Drift detection is off on operator-generated HelmReleases, so a capability gate in the templates would render the policy out at first install and never add it back."* Consequence: migration 50 (etcd adoption, v1.6.0) is fail-closed on the snapshot target. It resolves the target from the projected credentials Secret, which has no source without `<bucket>-<user>-credentials`, so its Job reboots forever and the cluster cannot be upgraded at all. ### 1. `backupstrategy-controller` reconciles the objects into existence New leader-elected `DefaultObjectsGate` runnable (`internal/backupcontroller/default_objects_gate.go`), modelled on the existing `SystemCredentialsProjector`. Every minute: - resolves the bucket name from the **projector's source Secret** (`parseSourceSecret`) — no new dependency, no `lookup`, and it covers the external-S3 path too. While it is unresolvable the gate does nothing: forcing then would only re-run the same empty lookup; - reads `BackupClass cozy-default` as the manifest of what must exist (its `strategyRef`s), plus the `cozy-default` BSL when `velero.bslEnabled`; - when any is missing, stamps `reconcile.fluxcd.io/forceAt` **and** `requestedAt` on its own HelmRelease. Both are required: `requestedAt` alone only requests a reconcile, which is a no-op for an unchanged release; `forceAt` is what triggers a real Helm upgrade, and it is only honoured together with `requestedAt`; - throttled to one forced upgrade per 5 minutes, and a no-op in the steady state. A `strategyRef` whose CRD is not installed is not counted as missing, so a missing CRD cannot drive an endless force loop. Two new metrics: `cozystack_backup_default_objects_missing{backupclass}` — the alert that would have caught this, since it is non-zero regardless of the HelmRelease's `Ready` condition — and `cozystack_backup_default_objects_force_reconciles_total`. RBAC delta is two lines: `patch` on `helmreleases` (`get` was already there) and `get` on `backupstoragelocations`. Both accesses are point Gets through the dynamic client, so no cluster-wide informer and no `list`/`watch`. Off switch: `backupStorage.reconcileDefaultObjects: false`. **Why not a render-time `fail` in this chart:** it would deadlock. `templates/bucket.yaml` in this same chart is the producer of the `Bucket` whose `BucketClaim` the lookup reads, so a failed render never applies it and the condition can never resolve. **Why not template the objects in Go:** their bodies are values-driven (endpoint, region, `forcePathStyle`, the Altinity strategy's whole PodTemplateSpec). Helm stays their single author; the gate only makes sure the render that produces them actually happens. **Why not `dependsOn`:** an edge guarantees the `Bucket` CR is *applied*, not that its claim is *reconciled*, so it does not close the race. ### 2. The gate also repairs the credentials Secret, one release earlier The `<bucket>-<user>-credentials` Secret is the same trap as the Strategy CRs, one release up the chain: `packages/system/bucket/templates/user-credentials.yaml` renders it behind a `lookup` of the COSI Secret, and skips it just as permanently. It is also the *first* domino — the projector reads it, so without it there is no `cozy-backups-creds`, no strategy, no Velero, and no migration 50. An earlier revision of this PR made that render `fail` so the release would retry and converge. [@IvanHunters pointed out](https://github.com/cozystack/cozystack/pull/3524#issuecomment-5214269078) that Helm cannot render a partial set, so the `fail` aborts the **whole** `<bucket>-system` release — the other users' Secrets and the bucket UI Deployment/Service/Ingress/HTTPRoute with them. One declared user whose `BucketAccess` never provisions would take down every other user of that bucket, and on an installed release park it in `Failed`, blocking every later upgrade. That was a real blast-radius regression on a universal path, and it is gone. The repair now lives in the gate, where it is per-object: - while the source Secret is absent, or carries no bucket name, the gate stamps `forceAt` + `requestedAt` on the bucket's `bucket-<name>-system` HelmRelease instead of doing nothing; - **no self-deadlock:** the `BucketClaim` and the `BucketAccess` whose COSI Secret that lookup reads are rendered unconditionally by the **parent** release (`packages/apps/bucket/templates/bucketclaim.yaml`), not by the one being forced; - **no RBAC delta:** `patch` on `helmreleases` is already cluster-scoped; - the two releases are throttled **independently**. They are forced in sequence on a bootstrap (credentials, then the objects the resolved bucket unblocks), so one shared timestamp would delay the second by a full `MinForceInterval` for no reason; - coordinates come from the chart under `provisionBucket`, and are omitted on external S3, where the Secret is admin-managed and no release renders it. The bucket chart keeps its partial render. Its comment now explains why the skip is not self-healing and why failing is the wrong lever, and a test renders the UI with *every* user unresolvable so a reintroduced `fail` fails the suite. **Scope this narrows, deliberately:** tenant buckets keep the pre-existing silent skip. The gate owns only the platform bucket, which is the cluster-breaking path. Generalising the repair to every bucket needs a controller that owns tenant buckets and is a separate change. Two more corrections from the same review: - `cozystack_backup_default_objects_missing` was only written on the happy path, so the state this PR exists to catch reported `0` and the documented alert never fired. The credentials Secret now counts in `missing`, so an absent or rotated-away Secret moves the gauge. On an API error the gauge is deliberately left alone rather than flapping, and a new `cozystack_backup_default_objects_check_errors_total` marks it stale; the doc pairs the two instead of overstating the gauge. `force_reconciles_total` gained `namespace`/`name`, since two releases can now be forced. - `forceHelmRelease` does a point `Get` and skips a release with `spec.suspend: true`. helm-controller ignores both annotations while suspended (`cozyhr suspend` sets exactly that), so the gate was re-stamping every `MinForceInterval` for the whole suspension and climbing the force counter — which the runbook attributes to a render that is not producing the objects. The skip is logged and not counted. ### 3. Corrected comments and docs `_helpers.tpl`, `backupclass-default.yaml`, `values.yaml` and `docs/operations/backup-classes.md` all claimed the skip self-heals on the HelmRelease interval, and the doc suggested a `flux reconcile` that does not re-render. They now say it does not, and why. The two-release manual recovery for already-affected clusters is documented, including the easy-to-miss part: the credentials Secret is rendered by the `bucket-<name>-system` release, not by `bucket-<name>`. ### Manual recovery for already-affected clusters Only needed on a version without the gate. A plain `flux reconcile helmrelease` does **nothing** — it does not re-render. ```bash # 0. Confirm the bucket name is resolvable first — forcing before that # just re-runs the same empty lookup. kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}' # 1. The credentials Secret. Rendered by the -system release, NOT by # bucket-cozy-backups — easy to miss, and the projector (hence every # strategy, Velero, and migration 50) has no source without it. ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \ reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite # 2. The Strategy CRs and the Velero BSL. ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \ reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite ``` ### Validation - `go test ./internal/... -count=1` — pass, including 15 cases for the gate: force on missing, no-op in the steady state, throttling, unmapped kinds ignored, BSL skipped when disabled, patch failure surfaced without advancing the throttle, absent HelmRelease tolerated, plus force-on-absent-Secret, force-on-empty-bucket-name, external-S3 reports-but-does-not-force, independent throttles, suspend skip on each of the two releases, gauge written on the unresolved path, and errors counter on a failed check. - `make helm-unit-tests` — pass, with 2 new suites: 3 cases for the bucket chart (partial render preserved, and the UI still renders with every user unresolvable — the guard against reintroducing the `fail`), 7 for the gate's env + RBAC wiring. Those include that `.Release.Name` and `.Release.Namespace` match the HelmRelease the operator generates (`releaseName` becomes the HR's `metadata.name`, and no `spec.releaseName`/`targetNamespace` is set, so the Helm release name equals the HR name), that the bucket release coordinates track a renamed bucket together with `systemSecretName`, and that they are unwired on `provisionBucket: false`. - `make generate` — no diff. - `make go-unit-tests rd-presets-check test-check-readiness migrations-target-check` — pass. - `make bats-unit-tests` — every suite passes except `hack/migration-seaweedfs-db-adopt.bats`, which needs a Docker daemon not available in this environment ("docker is required: these tests run the migrations inside alpine"). Unrelated to this diff. **Not verified — please read this before approving.** There was still no live-cluster run. What that gap covers is smaller than it was: dropping the chart `fail` removed the universal behaviour change on every bucket's install path, so there is no longer a render-blind failure mode whose convergence has to be observed. What remains is one mechanism — `forceAt` + `requestedAt` on an unchanged release — applied to two releases instead of one, argued from read code rather than an observed install: `Strategy.Name = RetryOnFailure` on operator-generated HelmReleases (`internal/operator/package_reconciler.go`), and `bucketName = "bucket-" + claim.UID` in `packages/system/objectstorage-controller/images/objectstorage/patches/92-bucketclaim-propagate-ready.diff`. It is the same procedure as the manual recovery below, which has been run by hand on the reporter's cluster. Someone with a bootstrap cluster should still confirm that the gate fires within a minute or two of the bucket becoming ready, and that forcing `bucket-<name>-system` materialises the credentials Secret. ### Screenshots Not applicable: no UI change. This PR touches a controller, two system charts, and docs. <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> Walked file by file against the diff of this PR. **`website` — affected, follow-up issue opened.** `content/en/docs/next/operations/services/backup-classes.md` mirrors this repo's `docs/operations/backup-classes.md` and carries the exact stale claim this PR corrects, plus a `flux reconcile helmrelease` recovery suggestion that has no effect. `operations/configuration/platform-package.md` enumerates the forwarded `backupStorage` keys, and `reconcileDefaultObjects` is admin-settable through `spec.components.platform.values.backupStorage` (the whole block is deep-merged), so that list becomes incomplete. Filed as an issue rather than a PR because the wording depends on the shape this PR lands in, and whether to touch the frozen `v1.5`/`v1.6` copies is a maintainer call. No existing website issue or PR covered it. **`terraform-provider-cozystack` — not affected.** Checked, not assumed: `packages/apps/**` is untouched by this diff (`git show --stat HEAD -- packages/apps/` is empty), so no app `values.schema.json`, no `values.yaml` default, no version enum, and no `ApplicationDefinition` `kind`/`plural`/`release.prefix` changed. `reconcileDefaultObjects` is a key on a **system** chart, which has no `values.schema.json` and is not modelled by the provider. (An earlier revision added `requireUserCredentials` to the `bucket` system chart; that is gone, and no key was added to `apps/bucket`, so the app API is untouched.) The `<bucket>-<user>-credentials` Secret the provider looks up by literal name is **not renamed** — this PR only changes whether it gets created, which if anything removes a case where the provider silently read null from a Secret that never existed. Everything else: no `hack/` layout or make-target change (`ccp`, `external-apps-example`, `cozyhr`), no `ApplicationDefinition` CRD change, no namespace rename, no node prerequisites in `hack/e2e-prepare-cluster.bats` (`talm`, `ansible-cozystack`), no `cozyhr.cozystack.io/values-files` annotation or chart-source-kind change, no `service-proxy-name` / `wholeIP` / `allowICMP` rename (`cozy-proxy`), no telemetry metric or label rename (`cozystack-telemetry-server` — the two metrics here are new, not renamed), no node or network requirement change (`examples`). - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/639 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(backupstrategy-controller): the default backup Strategy CRs, the Velero BackupStorageLocation and the bucket user-credentials Secret were skipped permanently when their install-time Helm `lookup` came back empty, leaving clusters with a BackupClass but no strategies and blocking the v1.6.0 etcd migration. The controller now detects the missing objects and forces the Helm upgrade that creates them, including on the bucket release that renders the credentials Secret. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added automatic detection and recovery for missing default backup resources. - Missing resources trigger a throttled Helm upgrade once bucket configuration is available. - Added configuration to enable or disable default-object reconciliation. - Added validation requiring user credential Secrets during bucket installation by default, with an offline rendering option. - **Bug Fixes** - Improved recovery from incomplete backup and Velero storage configuration. - **Documentation** - Expanded operational guidance, monitoring metrics, alerts, and manual recovery steps. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
66df3b20e0 |
docs(flux-shard-operator): drop unproven mechanism from proxy comments
Name all six proxy env spellings in the sanitisation list, and replace the unestablished 'external HTTPS call' / 'flux-aio does not restart' claims with the grounded in-cluster mechanism: after the KUBERNETES_SERVICE_HOST drop above, the shard talks to the management apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does not match, so that startup call is what stalls behind an unreachable proxy. Comment-only; no behaviour change. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
980715250c |
fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal (#3557)
## What this PR does The `kube-ovn-webhook` loaded its TLS serving certificate once at process startup (`tls.LoadX509KeyPair` into a static `tls.Config.Certificates`) and never re-read it. cert-manager renews the backing Secret ahead of expiry, but a running pod kept serving the certificate it loaded at boot. Once that certificate expired — roughly one year after install, on any cluster whose webhook pods had not been restarted after the renewal — the pod served an expired certificate, the kube-apiserver's TLS call to the webhook failed verification, and because the `MutatingWebhookConfiguration` uses `failurePolicy: Fail`, every pod creation in tenant namespaces was rejected. In practice this blocks `virt-launcher` pods, so no VMI can start. This PR: - Serves the certificate through a reloading `tls.Config.GetCertificate` callback that re-reads the key pair when the mounted files change, and keeps serving the last good certificate if a reload fails (e.g. a torn read while the volume is being updated). cert-manager renewals are now honoured without a pod restart. - Widens the certificate `renewBefore` from `24h` to `720h`, so renewal happens well ahead of expiry instead of one day before. - Adds a regression test that fails against the previous static implementation (the server keeps presenting the old certificate after the files are replaced) and passes once the certificate is reloaded; verified non-vacuous by mutation. ### Screenshots Not applicable (no UI changes). ### Downstream repositories - [x] No downstream repository is affected by this change Walked the trigger map file-by-file against the diff: the change is confined to the `system/kubeovn-webhook` image and its cert-manager `Certificate` `renewBefore`; it touches no package under `apps/`/`extra/`, no `values.schema.json`, no app `values.yaml` default, no `ApplicationDefinition`, no platform values, no CRD, and no `hack/` tooling. ### Release note ```release-note fix(kube-ovn): kubeovn-webhook now reloads its serving certificate when cert-manager renews it, so the webhook no longer serves an expired certificate that blocked all pod creation (including VMIs) roughly one year after install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Webhook TLS certificates now reload automatically when renewed, without requiring a server restart. * **Bug Fixes** * Certificate reload failures now fail the TLS handshake instead of serving an outdated certificate. * Added server timeouts to improve connection handling. * **Configuration** * Certificates are now renewed earlier to help prevent expiration-related service disruptions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
efa7f5c8fb |
fix(velero): apply CRD updates on upgrade via CreateReplace (#3727)
## What this PR does Velero CRDs stayed frozen at whatever version was first installed: Helm never touches the chart's `crds/` directory on upgrade, the generated HelmRelease left `spec.upgrade.crds` at the helm-controller default (Skip), and the package disables the upstream upgrade-crds Job. When the velero image moved to v1.18.1, which added the `Queued` and `ReadyToStart` backup phases, the apiserver rejected every phase transition against the stale CRDs (`status.phase: Unsupported value: "Queued"`) and backups silently stopped while the HelmRelease stayed green. This opts the velero PackageSource into `upgradeCRDs: CreateReplace`, the same policy mongodb-operator already uses, adds a platform test pinning it, and corrects the stale Job-disable rationale in the package. CreateReplace is safe here: velero only extends the phase enums between versions. Closes #3726 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(velero): apply Velero CRD updates on upgrade; previously CRDs stayed at their first-install version, so after the image moved to v1.18.1 backups could silently stop with `status.phase: Unsupported value: "Queued"` while the HelmRelease stayed green ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Velero now automatically creates or replaces its Custom Resource Definitions during installation and upgrades. * **Bug Fixes** * Improved Velero CRD lifecycle handling to ensure updated definitions are applied reliably. * **Tests** * Added coverage verifying the CRD upgrade configuration and updated validation guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f4d812cfb9 |
fix(velero): apply CRD updates on upgrade via CreateReplace
velero evolves its CRD schema between versions; v1.18 adds the Queued and ReadyToStart backup phases. Helm never upgrades CRDs shipped in crds/, the generated HelmRelease left spec.upgrade.crds at the Skip default, and the package disables the chart's upgrade-crds Job, so live CRDs stayed frozen at first-install state: the server upgrades, the apiserver rejects the new phases, and backups sit in New while the HelmRelease stays green. Opt the velero PackageSource into CreateReplace, consistent with mongodb-operator, add a platform test pinning the policy, and correct the stale Job-disable rationale in the package. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
f0bde76b9f |
docs(hack): fix a wrong claim about grep-error propagation in two contract tests
Both hack/promote-gate-contract.bats and hack/release-freeze-contract.bats carried the same code_lines helper with the same wrong comment: it claimed a real grep error fails the pin through the helper's own exit status. code_lines is never the last stage of a pipeline in either file, so ordinary (non-pipefail) pipe semantics discard whatever it returns on its own, and hack/cozytest.sh's translator appends a return 0 to every line that is exactly }, code_lines' closing brace included, making that exit status even less trustworthy as a signal. What catches a real grep error is the output, and only for one class of pin. The error empties code_lines' stream, so a pin that requires something to be present in that stream fails on it. A pin demanding an absence stays blind: an exact count of zero is what an emptied stream produces anyway, and what the pin was written to assert. Both files hold pins of that shape. The per-test [ -n "$block" ] guard is not the backstop either. In these two files a block is a bare awk extraction, upstream of code_lines, so that guard cannot see the filter fail; the comment now says so, because reading it as the backstop gets the direction exactly wrong. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com> |
||
|
|
ccd8a289ef |
fix(e2e): budget each linstor wait separately in post-install-prep (#3715)
## What this PR does Both waits in `hack/e2e-post-install-prep.sh` ran off one 15-minute deadline anchored at the script's start, so the head of the chain ate the window and the tail got whatever was left. Two runs on 2026-08-09 failed that way. The linstor HelmRelease went Ready 452s in, and the Deployment needed a further ~520s because its DB migration lost the apiserver and retried four times. Neither figure passes 900s alone. Their sum does, and the install reported the Deployment as broken. Each wait now gets its own budget, started when the previous link finished. Same shape as the node wait and the MetalLB CRD wait further down the same file. What it buys is that every link gets its whole allowance regardless of what the links ahead of it spent. The cost is on both sides and I measured it. Worst case goes from one 15m window to 900 + 900 + 300 + 300 = 40m, against the 180m the e2e job carries. A Deployment that never converges is now reported at 1355s instead of 900s, from the 452s above plus the loop's 5s poll. A HelmRelease that never goes Ready is still reported at 900s, which I checked by running the new suite against the pre-change script. The timeout now dumps the pods and recent events of `cozy-linstor`. The same "Deployment not Available" has come from a crash-looping migration init container and from a pod waiting on a client TLS Secret cert-manager had not issued yet, and telling those apart used to need the run's bundle. Events come from `kubectl events`, which sorts by last-seen and carries no `--sort-by` to fail on a key some item lacks. Both reads keep their stderr and are bounded, with the client budget smaller than the outer one so kubectl gets to say why before the kill lands. `hack/post-install-prep.bats` runs the script end to end against a stub PATH with a virtual clock, so a 15-minute budget elapses in a few hundred forks and the timings are exact. Four of its six tests fail against the pre-change script. The other two guard the bound that stays. ### Known gaps in the new tests Dropping the `2>&1` from either diagnostic read leaves all six tests green, because the harness catches the script's own stderr into the same file the assertion reads. The property holds in production anyway, since the caller merges both streams into one log, but nothing catches the removal. The test for "one failing read does not suppress the other" needs the pods read to run first; swap the two reads and it passes vacuously. Its comment says so at the point where someone would make that change. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the map against the diff. Two files, both under `hack/`. The `ccp` trigger is about moving or renaming things under `hack/` or changing what a make target does, and its anchors `hack/package.mk` and `hack/common-envs.mk` are untouched. `talm` and `ansible-cozystack` key off node prerequisites in `hack/e2e-prepare-cluster.bats`, a different file. ### Related Same family, found while working here and filed instead of folded in. #3707 and #3708 for unbounded reads elsewhere in this script, #3712 for the `--sort-by=.lastTimestamp` form in fourteen other places, #3713 for the wait loop dropping kubectl's stderr for its whole budget, #3714 for the MetalLB CRD wait failing without a message. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved LINSTOR readiness checks by giving each prerequisite its own 15-minute wait period. * Added clearer timeout diagnostics, including relevant pod and event information. * Ensured subsequent readiness checks are skipped when an earlier HelmRelease check fails. * **Tests** * Added end-to-end coverage for timeout handling, sequential failures, diagnostic errors, and readiness behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6ee56e3a60 |
fix(ci): decide E2E test coverage per changed path (#3441)
## What this PR does
`hack/select-e2e.sh` escalates to the full Chainsaw suite when a changed
package reaches no suite through the dependency graph, because otherwise
that change gets no E2E coverage at all. The escalation was read off the
merged selection, so it depended on the whole diff rather than on the
path that needed it. Any other changed path contributing a suite name
made the selection non-empty and the escalation vanished with nothing in
the output to say it had. Editing a platform component together with one
unrelated suite's Chainsaw test ran that single suite instead of
everything, which is an ordinary PR shape: change a component, adjust
the e2e near it.
Coverage is now decided per changed path. Each path resolves through its
owning sources to the suites that exist, and a path reaching none of
them forces the full run whatever the rest of the diff selected. The
unit is the path rather than the source, because
`system/postgres-operator` belongs to two PackageSources where one
reaches no suite, and deciding per source would run everything on every
change to it.
The other half is the mapping that left packages unreachable from their
own suites. `src_to_suites()` covered `*-application` names plus
`external-dns`, and the final filter dropped the rest, so
`cozystack.kuberture` and `cozystack.securitygroup-controller` were
walked to and then discarded. A CNI change ran neither of the two
controller suites most likely to regress from it, and a change to either
package escalated to the full run rather than selecting the suite that
covers it. Every reached source now maps through the table, which is the
inverse of `select-install.sh`'s `suite_to_source()`; a test walks every
suite through both, so the next source name fitting neither convention
fails there instead of quietly escalating forever.
## What was measured
Behaviour was diffed against the current selector over one `values.yaml`
per package under `packages/{apps,system,extra}`: 159 packages, 8
selections change, all of them the mapping repair. Five networking
packages and `prometheus-operator-crds` widen — 19 to 21 suites and 10
to 11, both gaining `kuberture` and `securitygroup`. `kuberture` and
`securitygroup-controller` narrow from the full suite to their own
single suite. Nothing narrows that should not.
That sweep is one path per diff, so it says nothing about the per-path
escalation, which only shows on a diff with more than one path. Pairing
each package with one unrelated Chainsaw edit gives that number. 82 of
the 159 packages escalate on their own and stop escalating once the pair
is added, which is the size of the hole: for those, adding an unrelated
suite edit to the same pull request replaced the full run with one
suite. 88 packages select something different after this change, the
extra six being the mapping repair widening a selection rather than
restoring an escalation.
Worth stating alongside it, because it decides whether the branch is a
corner or the common case: 83 of the 159 escalate on a single-path diff
today, and 81 do after this change — the two that leave are `kuberture`
and `securitygroup-controller`, which now select the suite that covers
them. So the escalation is the branch most packages take, and the
swallowed one was the branch most packages took. Counting that needs an
instrument that separates a graph walk which reached everything from an
escalation which printed everything, since the two agree once the
mapping is fixed: the count above comes from a scratch suite tree
carrying one extra suite that no source names, which only an escalation
can emit.
The suite selection is what decides which tests run, so a rule that
stops firing does not go red, it disappears. The tests pin the property
rather than a worked example: that a path reaching no suite escalates
whatever else the diff holds, that a covered path never drags the full
suite in with it, and that every suite round-trips between the two
mapping tables. Each was checked by mutation — removing the rule it
guards turns it red.
## Left for a follow-up
`resolve_suites` writes into the caller's scope because POSIX sh has no
`local`, and what keeps that safe is the single call site sitting inside
`$( )`, pinned by a test that counts occurrences of the name. Declaring
the function with `( … )` instead of `{ … }` would contain those
variables by construction and let both the test and the two paragraphs
explaining the constraint go away. A count-the-name grep is the weakest
kind of pin and it exists only because the function leaks.
The output contract at the top of the script lists the causes of a
full-suite selection but not the backstop, which after this change is a
mechanism of its own: an edit under `hack/e2e-chainsaw/<dir>/` where
that directory holds no `chainsaw-test.yaml`. The behaviour is pinned by
a test and explained where it lives; only the header block is short. In
the same area, the escalation line names the owning sources rather than
the changed path, and reports the first uncovered group rather than all
of them.
## Overlap
#3434 fixes the same escalation from a different angle: it decides the
escalation from the whole source-derived selection rather than per path.
That closes the "package with no coverage plus an unrelated suite edit"
shape, and leaves "package with no coverage plus another package that
does select suites", where the source-derived selection is non-empty and
the escalation is swallowed again. There is a test for that second shape
here.
### Screenshots
Not applicable, no UI changes.
### Downstream repositories
Walked the trigger map in `docs/agents/contributing.md` against the
diff. The changed files are `hack/select-e2e.sh`,
`hack/select-e2e_test.bats` and `docs/agents/e2e-testing.md`. The only
`hack/`-related trigger is cozystack/ccp, which gates on
`hack/package.mk`, `hack/common-envs.mk` and `hack/update-crd.sh` and on
what a make target does; nothing here is moved, renamed, or changed in
make-target behaviour. The ansible-cozystack and talm triggers are about
`hack/e2e-prepare-cluster.bats`, which is untouched.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(ci): a changed package that no E2E suite covers now runs the full suite even when the same pull request also edits an unrelated suite, and changes to the kuberture and securitygroup packages select their own suite instead of escalating
```
Closes #3330
Closes #3665
|
||
|
|
361c729417 |
test(codeowners): guard the coverage invariant with a unit check
CODEOWNERS is last-match-wins, so a new scoped rule that omits the default owners silently strips their ownership for that path. Add a hack/*.bats unit test (auto-discovered by make bats-unit-tests) that requires every rule listing owners to repeat the owners of the * catch-all, exempting ownerless rules and the governance roster. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
bd7b76691e |
chore(codeowners): keep architects as owners in every non-governance rule
CODEOWNERS applies only the last matching pattern, so a scoped rule replaces the * catch-all instead of extending it: @kvaps and @lllamnyp were not code owners for most scoped paths, and several paths (/hack/, /.github/, /Makefile, /.pre-commit-config.yaml, /docs/) were owned by exactly two people, deadlocking PRs authored by either of them while the other is away. Append @kvaps and @lllamnyp to every rule that lists owners, except the governance block which deliberately stays with @tym83 @kvaps, and spell out the non-additive last-match semantics in the file header. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
ba15b3817a |
fix(cozystack-basics): normalize tenant hostname VAP comparisons with lowerAscii
The Ingress, Gateway, and Route hostname ValidatingAdmissionPolicies compare tenant hostnames against the tenant apex with case-sensitive CEL. The resource-side host is already lowercase — the apiserver's hostname validation permits nothing else — but the apex operand is the namespace.cozystack.io/host label value, which Kubernetes permits to carry uppercase, and the Ingress policy additionally compares against the templated root host, a free-form platform value that nothing validates for case. A mixed-case apex label wrongly denies a tenant its own legitimate lowercase in-apex hostname. An uppercase root host is the dangerous direction: the Ingress outside-root exclusion compares a lowercase host against the uppercase literal, never matches, so the exclusion stops excluding and the policy degenerates into admitting any non-wildcard host — every non-wildcard platform-apex name a tenant asks for, including the root apex itself. Only the anti-wildcard gate still holds. Wrap both operands of every comparison in .lowerAscii() so the match follows DNS case semantics — no behavior change for the all-lowercase default. Each policy reads a label it does not own: the namespace host label is settable out-of-band by a trusted caller, so none of them may assume the apex arrives normalized. The Gateway policy admits any listener hostname when the host label is absent or empty, unlike its two siblings. That is the platform's one documented fail-open exception, recorded in the threat model together with its compensating control, so changing the posture is a security-model decision rather than a side effect of normalizing case. This leaves the behavior untouched and pins its current shape in the tests. Covered by helm-unittest assertions pinning each comparison as a whole operand pair, so dropping the normalization from any single operand is caught; a rendering case that pins the uppercase-root-host escape shut; and e2e probes that set a mixed-case apex on an isolated namespace and assert both halves of the match — a lowercase in-apex HTTPRoute and Ingress are still admitted, while a foreign apex and a hostname that ends with the apex without a dot boundary are still denied. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
86674c7084 |
fix(e2e): budget each linstor wait separately in post-install-prep
The post-install prep script waits out a serial reconcile chain -- the linstor HelmRelease, then the linstor-controller Deployment -- against a single 15-minute deadline anchored at the script's start. The head of the chain therefore eats the window and the tail inherits whatever remains, so an install fails whenever the chain merely runs long rather than never converging. Measured on one such failure: the HelmRelease went Ready 452s in, and the Deployment needed a further ~520s because the controller's DB migration lost its apiserver connection and retried four times before succeeding. Neither figure exceeds 900s on its own; only their sum did, and the install reported the Deployment as though it were broken. Give each wait its own budget, started when the link before it completed -- the shape the LINSTOR node and MetalLB CRD waits further down already have. What that buys is not speed but a guarantee: every link gets its whole allowance regardless of what the links ahead of it spent, so the tail can no longer be failed by the head being slow. It is a trade, and the other half is worth stating. The script's worst case becomes 900 + 900 + 300 + 300 = 40m against the 180m the job carries, where before the two waits shared one 15m window. And a link that is genuinely stuck is reported later rather than sooner: a Deployment that never converges surfaces at 1355s where the shared window surfaced it at 900s, on that run's timings. A HelmRelease that never becomes Ready surfaces at the same moment as before, since its budget still starts with the script. A timeout now also lists the linstor pods and the namespace events. The description names the object waited on, not the link that held it up, and the same "Deployment not Available" has been reached both with the controller pod crash-looping its migration init container and with that pod blocked on a client TLS Secret cert-manager had not issued yet. Telling those apart previously required the run's bundle. The events come from `kubectl events` rather than `kubectl get events`. It orders by when an event was last seen, so a condition repeating for minutes -- what a stuck mount looks like -- stays inside the window instead of being pushed out by newer one-off events, and it carries no --sort-by, so it cannot fail the whole read the way a sort key does when one item lacks it. Both reads keep their stderr and carry a bound. A read that fails silently is indistinguishable from a namespace with nothing to report, which is the distinction being drawn; and the caller is blocked in `wait`, so an unbounded read against a wedged apiserver would hold the install open until the job's own timeout. The client budget is strictly the smaller of the two so kubectl gets to name the reason before the outer kill takes it away. The wait helper is named for linstor because those diagnostics read one namespace unconditionally: the wait is generic over its kubectl arguments, what it dumps on the way out is not. The new unit test runs the script end to end against a stub PATH carrying a virtual clock, so a 15-minute budget elapses in a few hundred forks and the timings are exact rather than wall-clock approximate. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
587a7b9fbd |
fix(ci): decide E2E coverage per changed path
The full-suite escalation for a changed package that reaches no suite was
read off the merged selection, so it depended on the whole diff rather
than on the path that needed it. Any other changed path contributing a
suite name made the selection non-empty and the escalation vanished with
nothing in the output to record it: edit a platform component, adjust the
Chainsaw test beside it, and the component ran that single suite instead
of everything. Over one values.yaml per package under
packages/{apps,system,extra}, 82 of the 159 escalate on their own and
stop escalating once an unrelated suite edit joins the diff.
Coverage is now decided per changed path. Each path resolves through its
owning sources to the suites that exist, and a path reaching none of them
forces the full run whatever the rest of the diff selected. The unit is
the path rather than the source: system/postgres-operator belongs to two
PackageSources where one reaches no suite, so deciding per source would
escalate on that half and run everything for every change to it.
The escalation names its cause on stderr, as every other escalation in
this script already does. Most packages reach no suite of their own, so
this is the branch a full run usually takes; without a line saying which
sources bought it, "why did this run everything" has no answer in the log.
The walk moves into resolve_suites() to be called once per path. POSIX sh
has no `local`, so the function writes into the caller's scope and is
safe only inside the $( ) its single call site wraps it in; a test pins
that the call count stays one.
Two entries leave src_to_suites. `postgres-application` restated what
stripping the suffix produces and `external-dns` restated what carrying
the source name through produces, so both made the exception list read as
longer than the set of real exceptions.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
6322b49d74 |
test(qdrant): pin the tenant-visible Secret surface on both layers
Two objects decide which Secrets a tenant may read: the dashboard Role with its binding, and the ResourceDefinition. Adding qdrant-<name>-ca to either hands every tenant subject the CA private key — enough to issue certificates for anything — and nothing failed when that happened. Each layer is pinned whole rather than checked for known-bad names, because the ways this breaks add no name to notice: a rule granting secrets with no resourceNames restricts nothing, and a label selector can reach a key-bearing Secret while every name stays correct. The files carry the reasoning, including what the pinning does not settle. The ResourceDefinition is checked against the rendered chart rather than the file on disk, since the template globs a directory and a second file there ships the same way. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0980e755eb |
fix(ci): map E2E suites from sources not named after an app
src_to_suites() covered PackageSource names ending in -application plus external-dns, and the filter after the graph walk dropped every other source it reached. cozystack.kuberture and cozystack.securitygroup- controller are neither, so both were walked to and then discarded: a CNI change ran neither of the two controller suites most likely to regress from it, and a change to either package reached no suite at all and escalated to the full run instead of selecting the one that covers it. Every reached source now maps through the table. That table is the inverse of select-install.sh's suite_to_source(), and a test walks every suite through both, so the next source name fitting neither convention fails there rather than quietly making its own suite unreachable. The cilium test measured the gap rather than hid it, asserting the 19 suites the selector could reach and naming the two it could not. The subtraction is removed rather than inverted: kept as an exclusion list it would stay green if the walk stopped reaching them for an unrelated reason. What replaces it is a second assertion against a tree holding one suite the graph reaches and one no source names, which an escalation cannot satisfy. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
ebf2b2a08a |
feat(qdrant): publish ca.crt-only trust anchor to tenants
A qdrant release with TLS on mints a self-signed CA whose Secret also holds the CA private key, so tenants had no way to verify the server without being handed a signing key. The chart now declares a TenantProjection sentinel naming that Secret. The CA-extraction controller reads it and publishes a key-free copy, ca.crt only, as <release>.tenant-ca; the qdrant ResourceDefinition selects that projection by label so it surfaces in the tenant's secret list. No key-bearing Secret gains a tenant-visible label. The sentinel is gated on the chart's TLS condition. Unlike engines whose operator mints a CA unconditionally, qdrant only has one while TLS is enabled, and an ungated sentinel would sit Ready=False/SourceNotFound on every plaintext release. Tests pin the two invariants that fail silently: the sentinel carries no ownerReferences, so the lineage walk falls through to the Flux release label, and neither key-bearing Certificate stamps a tenant-visible label on its Secret. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
677d80a479 |
feat(kubernetes): route tenant worker image pulls through a registry mirror (#3575)
## What this PR does Refs #3513. It addresses one variant of that flake and does not close it; see "Scope against #3513" below. ### Root cause of the variant this addresses The chainsaw `kubernetes-latest` / `kubernetes-previous` suites fail with `node-join failed: fewer than 2 tenant nodes Ready within 18m` and zero registered tenant nodes. The in-guest Talos capture on #3548 shows where the time goes on one class of those runs: the worker loops trying to pull `ghcr.io/siderolabs/kubelet` from public ghcr.io and fails with `Head "https://ghcr.io/v2/siderolabs/kubelet/manifests/...": net/http: TLS handshake timeout host=ghcr.io`. The Talos `kubelet` service never starts (`kubelet was not registered`), so the node never registers, cilium-operator stays `Pending` for lack of a node to schedule on, and the budget expires. On those runs the tenant apiserver is healthy and the worker VMs boot with kube-ovn IPs, so the failing link is the worker's public image-pull egress, and the worker machine config has no `machine.registries.mirrors`, so there is no way to route around it. Same class as the Talos OS image, which #3231 solved with the `talos.imageFactoryURL` cache. The thread also carries a later analysis putting worker-VM CPU starvation from nested-virt oversubscription at the root, with serial-console RCU stalls showing the guest never reaching Talos userland, and reading the TLS-handshake variant as a symptom of the same contention. Nothing here contradicts that. Removing a live dependency on public egress from worker bring-up is worth doing on its own terms, and if the contention account is the dominant one, this narrows what is left to explain rather than competing with it. ### The change **Chart (`packages/apps/kubernetes`, `packages/apps/kubernetes-nodes`)**, a `talos.registryMirrors` passthrough, a map of upstream registry host to `{ endpoints: [ ... ] }`, rendered into `machine.registries.mirrors` of the worker `TalosConfigTemplate` that the MachineDeployment clones from, so it applies at first boot rather than after it. Empty by default: with the default values the rendered chart is byte-identical to main, both content-hash object names included, so no existing worker rolls. This is the durable half; an air-gapped, rate-limited or flaky-egress environment can point worker image pulls at its own mirror, which is the `machine.registries.mirrors` knob the Talos rollover left unfilled. Talos still falls back to the upstream registry unless a host also sets `skipFallback`, so a mirror on its own is not air-gap enforcement, and the field doc says so. The value lands inside the reconcile Job's unquoted heredoc, so it is escaped for backslash, dollar and backtick and renders as a literal. `talos.installerRepository`, `talos.schematicID` and the Talos version render into that same heredoc one block above it and now get the same treatment, so the file is consistent about it rather than escaping only the newest field. Escaping the rendered value covers every byte by construction, which enumerating known-bad shapes does not. The chain is a no-op for values carrying none of those characters, so the default render, content-hash Job name included, is unchanged and no existing worker is replaced. **e2e harness**, an in-sandbox `registry:2` pull-through cache for ghcr.io (`hack/e2e-ghcr-mirror.yaml`), applied at install time from its own step, with `hack/e2e-chainsaw/_lib/ghcr-mirror.sh` setting `spec.talos.registryMirrors` on the tenant CR when the mirror is up and its egress allow is in place, and emitting nothing otherwise so workers pull directly. A `CiliumClusterwideNetworkPolicy` opens one tightly-scoped hole for worker VM (`kubevirt.io: virt-launcher`) Pods in `tenant-test` to reach the mirror, mirroring the existing `talos-image-cache` policy. Two properties of the helper worth calling out, because both are easy to get wrong and both are now pinned by tests. A failed API call is never cached: the decision is reused by every later suite in the shared sandbox, so caching one blip would disable the mirror for the whole run. Readiness is decided once and acted on once. `rollout status` is the cheap way to learn it, but it also exits non-zero on a broken watch, so when it fails the Deployment's own `Available` condition is read rather than the watch's exit code taken as the answer. Exactly one failure is cached, a definite `Available=False`, and that is a trade rather than an invariant: it avoids re-paying the wait in every later suite, and costs a mirror that recovers mid-run. On a node-join failure the suite dumps the mirror's state alongside the Talos image cache it already dumped. Workers being pointed at the mirror and then joining does not establish that the mirror served anything, because Talos falls back to public ghcr.io on its own; the registry's access log is what separates the two, so the dump counts kubelet-image requests across the whole log rather than tailing it. The readiness probe writes an access line every five seconds, so by the time a run has spent its 18m budget a fixed tail window is hundreds of lines short of the request worth finding. ### Testing - `helm unittest packages/apps/kubernetes`, 205 tests, 22 suites; `helm unittest packages/apps/kubernetes-nodes`, 14 tests, 4 suites. New cases: the default omits `machine.registries.mirrors`, and a set value renders it with the endpoint. - `hack/ghcr-mirror_test.bats`, 23 tests. Beyond the manifest's two-phase split, egress identities, pull-through config and the pure YAML builder, this covers every outcome of `resolve_ghcr_mirror_endpoint` against a stubbed `kubectl`: an API NotFound caches, a transient query failure does not, a shell reporting a missing binary is not mistaken for either, a failed egress allow does not cache and its reason reaches the log, a rollout watch that failed for a reason the API does not confirm does not cache while a definite `Available=False` does, and a committed endpoint is cached and short-circuits the next call. The stub honours `--tail`, so the diagnostic's tests tell a filtered read from a truncated one instead of trusting the flag, and it reports errors on stderr as the real binary does. The suite also pins that the install step excludes the Cilium policy from its pre-Cilium apply, not only what that exclusion produces. - `hack/talos-reconcile-heredoc_test.bats`, 5 tests, which render both charts and run the heredoc through a real shell with hostile values rather than regexing the rendered string. A `matchRegex` cannot catch a heredoc the shell refuses to emit; this runs it. - `hack/run-kubernetes-talos-spec_test.bats`, 5 tests over `talos_spec_block`, the point where the two optional fragments are merged. The assertions splice the result under `spec:` and read it back with yq, so a block indented into the wrong parent fails instead of passing. - `packages/apps/kubernetes-nodes/tests/render-parity.sh` byte-identical; per-chart `make generate` in both charts with no drift; both `cozyrds` `openAPISchema` blocks equal to their `values.schema.json`. - The pinned `mirror.gcr.io/library/registry:2.8.3@sha256:a3d8aaa6...` is the digest Docker Hub serves for that tag, and `mirror.gcr.io` serves the same OCI index with a linux/amd64 manifest. Run under exactly the security context the Deployment sets, non-root 65532, read-only root filesystem, all capabilities dropped, writable state only on the mounted volumes, it starts, answers `/v2/` with 200, and serves `ghcr.io/siderolabs/kubelet` manifests and blobs as an anonymous pull-through. - `shellcheck` and `git diff --check` clean on the new and edited shell. ### What a CI run still has to answer Two runtime facts the harness half rests on, both recorded in the manifest: 1. Talos honours a plain-http mirror endpoint via `machine.registries.mirrors[ghcr.io].endpoints=[http://...]`. If a scheme or TLS knob is additionally needed, `machine.registries.config` has to be added too. 2. The containerd pull from inside the worker VM leaves with the virt-launcher Pod's own Cilium identity, which is what the policy selects. The label half is settled: `run-kubernetes.sh` already reads tenant worker Pods by `kubevirt.io=virt-launcher`, but whether the in-guest pull is subject to that Pod's egress policy at all is not. The second one bounds how strong a claim the harness can make. The gate is a `rollout status` plus an accepted egress-allow object, and neither observes the tenant-side datapath, so the fallback paths cannot make CI worse while the committed path is only as safe as that assumption. The in-repo comments say this rather than claiming the mirror can only ever help. ### Named improvements left out A tenant-scoped reachability probe, like the byte-level 206 check `talos-image-cache.sh` runs from a Pod carrying the consumer's own label, is what turns assumption 2 from an assertion into a measurement, and would also let the suite assert the mirror served the kubelet pull instead of the worker quietly falling back. Left out because a probe that does not exercise the tenant network path would pass while real workers stayed blocked, and the one that does is a change of its own size. The same gap has a cheaper edge: both the install step and the resolver key on the Deployment alone, so an apply that made the Deployment but not the Service still reads as success. That one costs little in practice, since the guest gets a fast NXDOMAIN instead of a dropped SYN, but the gate does check less than it commits to. `registryMirrors` could be a typed `map[string]struct{ endpoints []string }`, the way `nodeGroups` already is in this chart, so malformed input is rejected at admission instead of failing late inside Talos. The helm-unittest escape cases assert the `$` path; the backtick path is covered by the execution-level heredoc bats and the pure-backslash path by the escape chain's construction, so the security contract is fully exercised across the two layers. A backtick assertion at the helm-unittest layer would express it at the fast layer too. The escape chain is spelled out at five interpolation sites across the two reconcile-Job templates. A named helper (an `include` template) would make the escape a single reusable primitive and shrink the surface where the heredoc invariant can be forgotten; the current per-site form is correct and covered, so this is a readability refactor rather than a fix. Only `machine.registries.mirrors` is passed through. `machine.registries.config` is not, so a mirror behind a private CA or one needing credentials still has no knob, and the platform-wide `registries` value does not reach tenant workers. The chart's breaking-change note says so, and also says which knob covers which artifact: the mirror routes registry pulls, while the worker OS disk image arrives as a raw HTTP artifact under `talos.imageFactoryURL` and no registry mirror can touch it. The helper's failure model is worth revisiting as a whole rather than per call site. Every call it makes to the cluster has three outcomes (the world says no, the world says yes, the call could not be made) and a shell exit code carries two, so the third is re-derived at each site from a different signal: stderr text for a NotFound, a field value for readiness, the returned message for a failed apply. Since the resulting decision is cached for the whole sandbox, a misclassification is permanent for the run rather than a blip, which makes each of those sites load-bearing. A single prober answering `yes`/`no`/`unknown`, with the caching rule stated once over that answer, would move the question out of the branches; the current shape reaches the same behaviour but re-establishes it five times. Left out here because it is a visible rework of code this PR only extends, and it would need its own verification of every site; tracked in #3682. One nicety left in the same helper: the diagnostic reads the mirror log twice, once with `--tail=-1` to count kubelet-image requests across the whole log and once with `--tail=50 --prefix` for a bounded context tail. The second read exists for `--prefix`, which attributes each line to its pod; collapsing to one read drops that. The attribution is worthless while the mirror is a single replica and load-bearing the moment it is not, in the one artifact read after a failure, and the second read is itself wall-clock bounded, so it is left in place. ### Merge-order dependency (resolved) #3676 landed first and this branch is rebased onto it. `ghcr_mirror_diagnose` now runs in the node-join failure block under the same phase gate as its neighbours, and each of its reads is wall-clock bounded by the same `COZY_DIAG_READ_TIMEOUT`/`COZY_DIAG_READ_GRACE` the block validates, so no single read can hold the op open. It is placed after the guest captures and before the image-cache re-probe: the console evidence is irreplaceable and must not be starved of budget, and the dump is still cheaper than the Pod-creating re-probe, so the spend order puts it between the two. When the diagnostics phase runs out of budget the dump is declined out loud instead of attempted. An ungated dump would spend that time anyway and take it from the captures behind it, while the mirror's state is partly recoverable from the reads that already ran, so gating costs less than it saves. The known cost is #3686: cheap reads ahead of the gate can drain the whole budget on a viscous apiserver, and the dump is then declined in exactly the runs where the mirror is a suspect. That is a defect of the budget, not of this dump, and it is tracked there. ### Scope against #3513 #3513 asks for three things: capture the guest side, act on the evidence, and do not raise the 18m budget again. The capture landed in #3548 and this consumes its output; the budget is untouched. What this acts on is the TLS-handshake variant only. The same thread documents runs that issue no apid CSR at all inside the whole budget, time lost before Talos reaches the kubelet image, which no registry mirror can affect. So #3513 stays open after this merges, and a green suite afterwards is not evidence the flake is closed. The manifest header says so too, for whoever reads it next. ### Both tenant suites are covered `kubernetes-previous` selects the previous Kubernetes *minor* out of `packages/apps/kubernetes/files/versions.yaml`, `run_kubernetes_test 'keys | sort_by(.) | .[-2]'`, not a previously-released chart. Both suites build their tenant CR from the same in-tree chart through the same helper, so both get the mirror. ### Screenshots Not applicable; no UI changes. ### Downstream repositories Walking the trigger map in `docs/agents/contributing.md` against the diff: **cozystack/terraform-provider-cozystack is affected**. The provider is hand-written, and the map lists "add, remove or rename a field in an app's `values.schema.json`" and "change a default in an app's `values.yaml`" as triggers for its schema, model and expand/flatten pair. This adds `talos.registryMirrors` with a `{}` default to both the `kubernetes` and `kubernetes-nodes` schemas. No other repository matches a trigger. The website's managed-app reference pages regenerate from each package's `README.md` on a stable tag, and this only changes an existing package's README. I have not opened the provider follow-up and have not ticked the box, because a ticked box with no link claims work that does not exist. Leaving it to a maintainer to decide whether the provider models this field and who files it. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(kubernetes): add `talos.registryMirrors` to route tenant worker node image pulls through a registry mirror for air-gapped or flaky-egress environments ``` |
||
|
|
3ae1e10c92 |
docs(kubernetes): correct the registryMirrors field and README claims
State in the README that in-guest registry pulls are routable again via talos.registryMirrors, which knob covers which artifact (the mirror routes registry pulls; the worker OS disk image arrives as a raw HTTP artifact under talos.imageFactoryURL and no registry mirror touches it), and the limits: skipFallback is needed for air-gap, machine.registries.config is not passed through, and existing nodes keep their boot-time config until replaced. Trim the field doc comment. The regenerated schema, README tables and cozyrds openAPISchema follow from those sources. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9d4bcd8d66 |
fix(e2e): wire the ghcr.io mirror into the tenant install and node-join flow
Deploy the pull-through mirror from its own install step rather than folding it into the Talos image-cache step, whose early return on a missing values key would otherwise skip it silently. Build the tenant CR spec.talos block from both the OS image cache and the registry mirror, each included only when its in-sandbox backend is up. On a node-join failure dump the mirror's state alongside the Talos image cache, under the same phase gate as its neighbours so it cannot starve the guest captures of budget. Record the mirror's legs in the step's budget inventory. talos_spec_block is covered by a bats suite over all four mirror-up and mirror-down combinations, splicing the result under spec: and reading it back with yq so a block indented into the wrong parent fails, and the install step is pinned to exclude the Cilium policy from its pre-Cilium apply. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
1c6ee7c047 |
fix(e2e): harden the ghcr.io mirror resolver and bound its diagnostics
resolve_ghcr_mirror_endpoint decides once, for the whole shared sandbox, whether tenant workers are pointed at the in-sandbox mirror, so a misclassified transient failure would disable it for the rest of the run. Cache only stable answers: a genuine API NotFound and a definite Available=False; never a transient query error, a missing kubectl, or a failed egress-allow apply. Read readiness from the Deployment's own Available condition when the rollout watch exits non-zero, and commit the endpoint only once the egress allow is in place, surfacing the apply failure reason rather than a generic blip. The node-join diagnostic counts kubelet-image requests across the whole access log rather than a fixed tail, and every read it makes is wall-clock bounded by the caller's COZY_DIAG_READ_TIMEOUT so no single read can hold the failure path open. The bats suite exercises every resolver branch and the diagnostic's filtering, with a kubectl stub that reports errors on stderr as the real binary does and a timeout mock so the bounded reads stay reachable under test. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
3a1292d682 |
fix(kubernetes): escape tenant values in the worker reconcile heredoc
The worker TalosConfigTemplate is applied from an unquoted `cat <<EOF | kubectl apply` heredoc, so every free-form value interpolated into it is subject to shell parameter expansion and command substitution at Job runtime. Escape backslash, dollar and backtick on talosVersion, installerRepository, schematicID and the registryMirrors block so each renders as a literal, and record the invariant above the data block in both charts: a new free-form field here must be escaped or render-time pattern-validated, or it reopens command injection into a host-cluster Pod. Escaping covers every byte by construction; the chain is a no-op for ordinary values, so the default render and its content-hash Job name are unchanged. Tests render both charts and run the heredoc through a real shell with hostile values, which a rendered-string regex cannot do, and the default-omits-mirrors assertion selects the Job by kind rather than by document index so it cannot pass against the wrong document. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
02a3fda471 |
test(e2e): capture host-netns address attribution and a healthy-pod reference for the dataplane collector (#3617)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Extends `hack/e2e-capture-dataplane.sh`, the diagnostic collector that runs after an e2e test failure, with the instrumentation issue #3615 asks for to close its address-attribution gap. Issue #3615 found that a NotReady pod's IP can turn out to be a local address of the node's host netns instead of reachable through the ovn0 route, which makes kubelet's probes connect to the node's own loopback and fail forever. The collector already captured `ip addr show ovn0` for the affected pod's node, but nothing that could name which interface actually holds a stolen address, since it only ever looked at ovn0. This adds `ip -o addr show` (every interface) and `ip route show table local` to the same host netns capture, so the interface actually holding a local address can be identified on the next occurrence. The collector also only ever inspected NotReady pods, so a run had no same-window baseline for what a normal route and conntrack state look like on the same node. This extracts the existing per-pod capture into a reusable function and runs it a second time against one Ready, Running pod on the same node as an in-run healthy-pod control, picked while explicitly excluding hostNetwork pods (their podIP is the node's own address, so one would show the exact same local-route fingerprint as the bug for an unrelated reason). When no eligible Ready pod exists on the node, that absence is recorded in the artifact instead of left silent. This is diagnostic tooling only: it changes what gets captured after a failure, not any test's pass/fail outcome. ### Known limitations The baseline runs at a reduced scope: the host route and the kernel conntrack for the healthy pod's address, and nothing else. What it drops -- cilium CT, the OVN port binding, the OVS `ovn-installed` flag, and the pod's own Ready conditions and events -- describes how a pod was programmed, and a pod picked for being Ready and Running is programmed correctly by definition. That scope is what holds the pod path's worst case at 579s against the 600s the `cozytest.sh` trap allows; the remaining 21 seconds are what this path leaves for the LoadBalancer section that runs after it, not headroom the run has. The other caller's envelope is 300s, and the worst case does not fit it -- nor did it at the merge base, where the same walk bounds at 519s. That overrun is tracked in #3671, and the term-by-term arithmetic is in the `MAX_PODS` note in the script. The 579s above assumes the lookup memo exists. When `mktemp` fails the memo is empty, every per-node pod lookup runs live again, and the same single-pod case bounds at 667s -- past the 600s envelope as well as the 300s one. At the merge base that memo-less case bounds at 585s, inside 600s, so this change is what moves it out. The degradation is visible rather than silent: the caller's backstop truncates the capture and the run reports `data-plane capture INCOMPLETE (exit 124)`. A baseline can be present and still carry nothing. Every capture at that scope goes through the node's `kube-ovn-cni` pod, so when that lookup does not resolve, the reference section is its banner and the scope note with nothing beneath them -- and, unlike `node-<node>.txt` next to it, no line saying why. Two different standards of honesty inside one artifact. The affected-pod path has the same shape today, so this extends an existing gap rather than opening one, and it is left alone here. The per-node baseline is cached under `TMPDIR` for the run. A SIGKILL -- which is what a caller's wall-clock backstop delivers -- skips the cleanup at the end of the script and leaves that directory behind. The lookup memo this script already used has the same exposure. Nodes are visited in the order their affected pods appear in the cluster-wide list, which the API chooses. Once `MAX_PODS` cuts the walk, which pods and therefore which nodes got captured is a function of that ordering and not of anything a reader would pick. ### Screenshots Not applicable, no UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note test(e2e): capture host netns address attribution and an in-run healthy-pod reference for the dataplane failure collector ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added diagnostics for node addresses and local routing information. * Added healthy, same-node reference captures for dataplane troubleshooting, reused across affected pods. * Expanded pod diagnostics to include status, events, connection tracking, routes, network bindings, and interface state. * **Tests** * Added coverage for pod selection, retries, caching, exclusions, incomplete records, unavailable responses, and missing baselines. * Recorded diagnostic failure reasons in artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
18be4ee272 |
test(e2e): capture host-netns address attribution and a healthy-pod reference
The collector characterises the host->local-pod route for an affected pod through ovn0 alone, so when a pod IP turns out to be a local address of the host netns instead, nothing records which interface actually holds it. The host netns capture now also takes `ip -o addr show` across every interface and `ip route show table local`, so the owner can be named on the next occurrence rather than only ruled out on ovn0. The collector also inspects NotReady pods only, so a run has no same-window baseline for what a normal route and conntrack state look like on the same node. The per-pod capture becomes a reusable function and runs a second time against one Ready, Running, non-hostNetwork pod on that node, into the same file, so the comparison is in front of whoever opens it. hostNetwork is excluded because such a pod's podIP IS the node's own address and would show the same local-route fingerprint as the fault for an entirely ordinary reason. That baseline depends on the node and nothing else -- the candidates are the rows of the one snapshot the affected set was computed from, and a pod reporting Ready there is by definition not one this run is investigating -- so it is captured once per node and folded into each affected pod's file from a cache. Several affected pods on one node is what a wedged host->local-pod datapath looks like rather than an edge case, and capturing the same healthy pod once per affected pod would spend the caller's envelope re-asking a question already answered -- the cost the node-global dedup and the lookup memo beside it exist to avoid. This adds no read of its own. The cluster-wide pod list gains one column instead of being asked a second time, and that single read's status is what lets the artifact separate a node with no eligible pod from a list that never answered: one line for both would make an unchecked baseline read as an established absence. The two host-netns additions are best-effort execs, like the captures they sit beside. The new row filter carries the field count its jsonpath emits, so a list cut mid-record cannot hand over a fragment as the pod every reading of the affected one is compared against. The unit suite pins these on what they are rather than on what they leave behind. Both host-netns captures are asserted through the kubectl argv the collector issues, container included, because the section headers are separate echoes that outlive the exec being removed. The baseline is pinned from both sides: several affected pods on one node share a single capture, and an affected pod is served the healthy pod from its own node rather than from whichever node the walk reached first. The pod list's column order is asserted against the positions its readers index, because every runtime test answers that list from a stub written by hand and none of them can observe the jsonpath at all. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |