admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-03 10:37:21 +09:00
Code Issues Packages Projects Releases Wiki Activity
5,813 Commits 540 Branches 294 Tags
kafka-multi-instance
Commit Graph
132 Commits
Author SHA1 Message Date
Alexey Artamonov a8863ee145 fix(kafka): harden migration hook and isolate the migration e2e 
Addresses the second review round on the ZK->KRaft migration.

Migration hook:
- Fully qualify the Strimzi resources as kafkas.kafka.strimzi.io /
  kafkanodepools.kafka.strimzi.io. The bare plural "kafkas" also exists in this
  chart's apps.cozystack.io group, so a bare name resolves by per-cluster
  discovery priority and can hit the tenant kind — which the hook Role does not
  grant, failing the pre-upgrade hook deterministically.
- Classify NotFound structurally via `kubectl get --ignore-not-found` instead of
  substring-matching stderr, so an unrelated "... not found" error can no longer
  be mistaken for a fresh install and permanently skip a live ZooKeeper cluster.
  This also drops the jq dependency.
- Treat a KRaft-native / already-migrated CR (annotated strimzi.io/kraft=enabled)
  as done, never driving it backwards into kraft=migration.
- Keep only `before-hook-creation` on the Job's hook-delete-policy so a failed
  attempt's pod logs survive for a post-mortem until the next run.
- Route the kubectl image through cozy-lib.image (cluster registry mirror) and
  pin it by digest (clastix/kubectl:v1.32@sha256:...), matching the other
  cozystack hooks, instead of an unpinned docker.io tag.

Controller quorum:
- Fix the controller node-pool replica count at creation instead of deriving it
  from kafka.replicas on every reconcile. Strimzi 0.45 cannot scale a controller
  pool, so a day-2 edit of kafka.replicas (e.g. 1 -> 3) must not flip the quorum;
  the helper now keeps the existing "controller" pool's replica count when it is
  present.

Docs:
- Document that migration to KRaft is one-way: neither a version-field downgrade
  nor a pre-KRaft chart rollback is supported once migrated (Strimzi has no
  KRaft->ZooKeeper rollback after completion).

E2E:
- Run the ZK->KRaft chainsaw migration in its own namespace instead of the
  shared tenant-test. It raw-applies Strimzi resources plus a namespace-wide
  LimitRange and leaves deleteClaim:false PVCs, which polluted the sibling kafka
  suite. Raise the seed readiness timeout for CI headroom. Validated against a
  live cluster: migration adopts the broker PVC in place and reaches KRaft under
  a 128Mi LimitRange, in isolation.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-08-26 18:25:58 +03:00
Alexey Artamonov 37bfd448ca fix(kafka): set resources on the migration node pools 
The ZK->KRaft pre-upgrade hook created the adopting broker ("kafka") and
"controller" KafkaNodePools without a resources block. Those hook-created pools
govern the Kafka pods for the entire migration window — Helm applies the
resourced steady-state kafkanodepools.yaml only after the pre-upgrade hook
finishes. In a tenant namespace the LimitRange defaults every container to
128Mi, so the migration's controller (and rolled broker) JVMs OOM, never reach
KRaftPostMigration, and the hook times out — wedging the migration and thrashing
it on every reconcile.

Give both hook pools the same resources the steady-state pools get
(cozy-lib.resources.defaultingSanitize from kafka.resourcesPreset/resources).
This also avoids a redundant second broker roll (resource-less -> resourced)
after the hook completes.

Tests: a helm-unittest assertion that both migration pools render a resources
block; the Chainsaw ZK->KRaft suite now seeds under a 128Mi LimitRange (and
gives the seed cluster explicit resources), reproducing the tenant-quota
condition, and asserts the migration still reaches KRaft with the broker PVC
adopted. Validated green against a live cluster under the LimitRange.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-08-26 18:25:58 +03:00
Alexey Artamonov 42e0c3dd84 fix(kafka): enforce one Kafka per namespace; simplify node pool names 
Follow-up to the KRaft-migration review. The broker KafkaNodePool must be named
exactly "kafka" so Strimzi adopts an existing ZooKeeper cluster's
<cluster>-kafka-N brokers and data in place during migration, but KafkaNodePool
object names are unique within a namespace — so two Kafka clusters cannot both
own a "kafka" pool there. This is an upstream constraint and Strimzi's own
guidance is one Kafka cluster per namespace
(strimzi/strimzi-kafka-operator discussions/11120).

Adopt that model instead of the earlier fresh-vs-migrated conditional naming:

- New kafka.assertSingleInstance helper fails the render (with a clear message
  and the upstream reference) if the namespace already holds another Kafka
  cluster. Reconciles/upgrades of the same release are allowed; lookup returns
  nothing during `helm template`, so unit tests never trip it.
- Node pools are now named "kafka" (broker) and "controller" unconditionally,
  which also adopts existing brokers on migration and yields clean
  <cluster>-kafka-N / <cluster>-controller-N names. The fresh-vs-migrated
  brokerPoolName helper is removed.
- WorkloadMonitor pool-name selectors and the migration hook updated to match;
  unit tests updated for the fixed pool names.
- README documents the one-Kafka-per-namespace rule and links the Strimzi
  discussion.

The ZK->KRaft Chainsaw migration suite (seed ZooKeeper cluster, migrate, assert
the original broker PVC survives and no wrongly-named PVC is created, cluster
reaches KRaft) was run green against a live cluster.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-08-26 18:25:58 +03:00
Alexey Artamonov e00c8f4a30 fix(kafka): address KRaft-migration review — broker pool adoption, hook safety, tests 
Addresses the branch review on the ZK->KRaft migration path.

CRITICAL — broker node pool adoption. Strimzi derives broker/PVC names as
<cluster>-<pool>-<id>; a pre-node-pool ZooKeeper cluster has brokers
<cluster>-kafka-N, so the adopting broker pool must be named exactly "kafka"
or the operator provisions fresh empty brokers and orphans the tenant's data.
A new kafka.brokerPoolName helper resolves the broker pool to "kafka" for an
existing (migrating) cluster and keeps the release-scoped "<release>-broker"
for fresh installs, so multiple Kafka instances still coexist in one namespace.
The migration hook creates the pool as "kafka" and fails closed if a "kafka"
pool already belongs to another cluster (node pool names are namespace-unique,
so in-place migration of co-namespaced ZK clusters is unsupported). Wired
through kafkanodepools.yaml, workloadmonitor.yaml and the hook.

MAJOR — migration robustness.
- Raise the HelmRelease timeout: the migration Job can run tens of minutes but
  the release timeout defaults to 10m, so helm-controller aborted and thrashed
  the pre-upgrade hook. Add release.cozystack.io/helm-install-timeout=45m on the
  kafka ApplicationDefinition, above the Job's activeDeadlineSeconds (2400s).
- Idempotent migration: do not re-issue kraft=migration once the cluster is past
  the ZooKeeper phase (an invalid backward transition Strimzi rejects on retry).
- Fail closed on the decision gates: distinguish a genuine NotFound (skip) from
  any other kubectl error (abort), and never treat an empty kafkaMetadataState on
  an existing CR as "migrated".

MINOR — version downgrade documented as unsupported; integer (not lexical)
compare in the version gate; deleteClaim/PVC-retention and the one-per-namespace
migration limitation documented in the README; docs/storage-immutability.md
updated for KRaft.

Tests — helm-unittest coverage for the broker pool name, the hook guards and the
idempotent/fail-closed logic; a bats check for the helm-install-timeout budget
ordering; and a Chainsaw ZK->KRaft migration suite that seeds a ZooKeeper
cluster, drives the migration, and asserts the original broker PVC survives (not
recreated) and the cluster reaches KRaft. The migration sequence and the
Chainsaw assertions were validated against a live cluster.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-08-26 18:25:58 +03:00
Matthieu b5b9525342 feat(kafka): migrate from ZooKeeper to KRaft mode with automatic migration 
Replace ZooKeeper-based Kafka with KRaft (Kafka Raft) metadata mode.
Strimzi 0.45 is the last line that still supports ZooKeeper, so this
prepares the transition before ZooKeeper support is removed upstream.

Chart changes:
- The Kafka CR always renders strimzi.io/kraft=enabled with node pools
  and no spec.zookeeper block. Storage, replicas and resources move to
  separate broker + controller KafkaNodePools (a homogeneous separate-pool
  layout works for both fresh installs and migrations; Strimzi 0.45 does
  not support migration into a combined pool).
- Kafka version is now user-configurable via the standard files/versions.yaml
  pattern (v3.9 default, v3.8 available); appVersion bumped to 3.9.0.
- WorkloadMonitors track the broker and controller pools (no zookeeper
  monitor); the dashboard RBAC grants the controller monitor and drops the
  zookeeper one.

Migration of existing ZooKeeper clusters is handled by a Helm pre-upgrade
Job (templates/migration-hook.yaml) gated by a <release>-kafka-deployed-version
ConfigMap, following the established seaweedfs/etcd hook pattern. The Job
creates the broker + controller pools with Helm ownership labels, drives the
Strimzi ZK->KRaft state machine (node-pools=enabled, kraft=migration ->
kraft=enabled once KRaftPostMigration is reached) and is skipped once the
ConfigMap records version >= "1".

Testing:
- Helm unit tests cover the Kafka CR, node pools, migration hook, metrics,
  topics, version map, dashboard RBAC and workload monitors.
- The e2e suite is ported to the current Chainsaw framework
  (hack/e2e-chainsaw/kafka): it asserts the KRaft topology (both node pools,
  kafkaMetadataState=KRaft), the absence of ZooKeeper, and bootstrap
  endpoints.

Rebased onto current main: TLS listener support, entity-operator resource
guards, x-cozystack-options schema annotations and the WorkloadMonitor
replica-path fix are all preserved and reconciled with KRaft.

Signed-off-by: Matthieu <matthieu@hidora.com>
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-08-26 18:11:57 +03:00
Timofei Larkin f5b957ee27 feat(kubernetes)!: manage worker pools as KubernetesNodes, adopt on upgrade (Phase 2, breaking) (#3315) 
## What this PR does

Phase 2 of the Kubernetes-app split
([cozystack/community#8](https://github.com/cozystack/community/pull/8))
— the **breaking** half. Stacked on #3314 (review that first).

Removes worker node pools from the `kubernetes` chart (they moved to
`kubernetes-nodes` in #3314) and adopts existing pools on upgrade:

- `kubernetes` chart is now **control-plane only**: drops the pool
`range` block, the `talos-reconcile` Job + RBAC (deleted; moved to
`kubernetes-nodes`), the preserve-old-KMT block, the
`kubernetes.nodeGroups` helper, and the per-group dashboard RBAC.
- **`spec.nodeGroups` is removed from the `Kubernetes` CR** (BREAKING).
Worker pools are now separate `KubernetesNodes` resources.
- **Migration 54** (bumps `targetVersion` to 55): pre-upgrade hook that,
for every `kubernetes-<cluster>` release, creates one
`kubernetes-nodes-<cluster>-<pool>` HelmRelease and re-annotates the
existing worker objects onto it with `helm.sh/resource-policy=keep`, so
the now control-plane-only parent upgrade does not prune running
workers. The default `md0` (implicit when `nodeGroups` was empty) is
materialised explicitly. Idempotent. Genuine infrastructure/read
failures fail closed (`exit 1`, retried on the next upgrade); hostile
stored shapes (a pool name over Helm's 53-char release-name cap, an
RFC-1123-invalid or whitespace/slash map key, a non-object `nodeGroups`)
are warned, pinned, and skipped rather than deadlocking the entire
fleet's upgrade.
- The `ingressNginx` addon no longer auto-provisions a node (the
implicit `md0` default is gone). If enabled without a pool carrying
`roles: [ingress-nginx]`, the controller DaemonSet simply schedules
nothing (degraded, not failed) — the requirement is documented in the
chart NOTES.txt rather than enforced at render time.

Because the child chart renders the pool objects **byte-identically** to
the old parent render (golden parity in #3314; the
KubevirtMachineTemplate content-hash name is preserved), adoption does
not churn live worker VMs.

### Migration number

`54` is contiguous with main (latest is `53`) and `targetVersion` is
`55`. The `52`/#3201 collision noted in earlier revisions has been
resolved: this branch was rebased and merged past it, so the numbering
is contiguous and `run-migrations.sh` has no gap to hard-fail on.

### Not yet done (blocking draft → ready)

- [ ] **dev e2e upgrade validation** (`cozystack-pr-test`): upgrade a
custom-`nodeGroups` cluster and a default-`md0` cluster; assert zero
worker-VM churn (KMT hash, Machine uid/creationTimestamp stable) and
that the adopted child HRs reconcile.
- [ ] downstream trigger-map walk (website docs for the `nodeGroups` →
`KubernetesNodes` change; terraform-provider for the removed API field).

### Downstream repositories

Deliberately unticked (draft) — the breaking API change affects docs and
the terraform provider; follow-ups to be filed before ready.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:

```release-note
Worker node pools of a managed Kubernetes cluster are now managed as separate `KubernetesNodes` resources. `spec.nodeGroups` on the `Kubernetes` CR is removed; existing pools are adopted automatically on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Worker pools are now separate `KubernetesNodes` resources with their
own lifecycle.
* Added a render-time compatibility guard to prevent worker pools from
running ahead of the parent control-plane minor version.
* Improved upgrade/uninstall behavior to better preserve already-adopted
worker objects.

* **Breaking Changes**
* Removed per-pool `nodeGroups` and worker node health settings from the
`Kubernetes` chart/CR; pool configuration now lives in
`KubernetesNodes`.
* Ingress requires at least one `KubernetesNodes` pool with `roles:
[ingress-nginx]`.
* `cluster` and `storageClass` are immutable after creation; pool
`version` may lag but must not be ahead of the parent minor.

* **Documentation**
* Updated READMEs, notes, and schemas to reflect the `KubernetesNodes`
split and the immutability/version rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-26 12:06:18 +04:00
Aleksei Sviridkin ad62a8d62b test(e2e): stop blocking merges on the tenant node-join deadline 
The tenant node-join wait no longer discriminates. Nested
virtualisation on the shared runners degrades far enough that a guest
which registers in minutes on real hardware can miss any deadline this
test can afford, so a red on that one wait says as much about the
machine the run landed on as about the product, and it teaches a
reader to re-run the job.

The suite still fails on it. That is what keeps Chainsaw's catch
running: the previous-instance container logs, the host snapshot, the
data-plane capture and the event dump are all keyed on the test
failing, and the report stays honest. What changes is one layer up.
The wait writes a marker beside its diagnostics, and a new lane-side
gate reads that marker to decide whether the job blocks. It opens only
when every failed test in the run carries one, and blocks on anything
it cannot attribute: a report it could not read, a failed test the
report does not name, a failure with no marker.

Only the lanes that block a merge ask it. The two pull-request lanes
do; nightly and the tag lane run the same suites and stay hard. A soft
pass on the tag lane would ship a release candidate whose tenant
clusters brought up no worker nodes, and nightly is the only place an
e2e failure reaches anyone who did not open a pull request, since no
lane in this tree notifies anywhere.

A softened red finishes the job green, so the job's result stops
answering whether the suite passed, and both kinds of consumer are
handed the decision instead of inferring it. A step gated on the job
failing would silently stop running: the SSH breakpoint is that case,
and it still opens for a pull request marked for debugging. A job that
reports the result would silently start reporting a pass: the required
commit status now says the deadline was missed and that the merge is not
blocked on it, rather than that the suite passed.

The deadline goes to 29m. That is the test's own budget and nothing
more: its clock starts after the machinedeployment wait, the
LoadBalancer assignment and the healthz probe, while the chart's
autoscaler and health-check timers have been running since the Machine
was created, so no ordering against them is claimed. The chart's 20m
and 30m give a scale to pick against, and a budget over the first
leaves room for a remediation cycle without promising that one fits.

The operation ceiling both suites give the script rises with the
deadline, and the diagnostics phase budget rises inside it, because
the collectors gated ahead of the serial console cost more between
them than the old budget covered. Both budget guards read their terms
from the sources that set them. The e2e job cap rises with them, and
its guard holds what a guard can hold: the pair's operation and catch
ceilings, and one figure across every lane that runs the suite. It
also records that the suites' full summed ceilings have never fit any
cap in that file, so the margin above the bound is taken from a
measured run instead. The prose that restates the cap elsewhere moves
with it, and the sweep that holds the deadline's restatements now
reads the figure spelled in words as well as in minutes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-21 12:52:42 +03:00
IvanHunters f035da8a0f Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Resolve conflicts introduced by main:
- Drop the FluxCD tenant addon (API type, deepcopy, values schema, RD) in
  favour of the GPU type main added; main removed the fluxcd addon wholesale.
- Renumber the Phase 2 worker-pool adoption migration 54 -> 56, behind main's
  redis-operator (54) and fluxcd-removal (55) migrations; targetVersion 55 -> 57.
- Accept the branch's removal of the in-chart talos-reconcile Job; it moved to
  the kubernetes-nodes chart, main only touched its comment.
- Regenerate kubernetes values.schema.json / README and the kubernetes-rd
  ApplicationDefinition schema.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-20 14:04:32 +03:00
Andrei Kvapil 7cca3d75e7 feat(kubernetes)!: remove the FluxCD tenant addon 
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a
Flux Operator + FluxInstance into a tenant cluster. It was served by two
vendored charts, packages/system/fluxcd-operator and packages/system/
fluxcd, which were the last AGPL-3.0 content in this Apache-2.0
repository and a governance blocker for CNCF Incubation.

The management cluster does not use these charts (it runs classic Flux
via internal/fluxinstall; migration 21 already removed the operator from
it), and the addon defaulted to disabled. Rather than maintain an AGPL
payload for an unused feature, drop it entirely:

- delete both system packages and their platform source entries
- remove the addons.fluxcd HelmReleases, values and generated schema
  from the Kubernetes app
- drop the addons.fluxcd block still submitted by the e2e harness
- drop the removed fluxcd package entries from CODEOWNERS

BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is
removed. Tenant clusters that had it enabled keep their running Flux
release; see the companion migration for the upgrade path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-20 13:34:32 +03:00
Aleksei Sviridkin 4385990869 feat(redis-operator): migrate from archived spotahome to freshworks-oss fork (#3406) 
## What this PR does

`spotahome/redis-operator` was archived on 2026-06-11 (read-only; last
release v1.2.4, 2022). This migrates the operator to the
actively-maintained **freshworks-oss/redis-operator** fork (v3.3.5,
Apache-2.0).

### Why freshworks-oss

It keeps the same `RedisFailover` CRD and API group
(`databases.spotahome.com`) — so existing managed Redis is unaffected —
and adds a native `spec.engine: Redis|Valkey` switch, letting managed
Valkey drop its Redis-compatibility shim image and run the official
`valkey/valkey` image directly.

### Changes

- Re-vendor the chart from
`oci://ghcr.io/freshworks-oss/charts/redis-operator` (v3.3.5), which
carries the `engine`-aware CRD.
- Build the operator image from freshworks-oss v3.3.5 source.
- Re-target the label/annotation-preservation patch onto the freshworks
Go module (`github.com/freshworks/redis-operator`).

### Verification

- freshworks-oss source + our `labels.diff` patch compiles natively with
Go 1.26.3 (`service` and `cmd` packages).
- `git apply --check` of the re-targeted patch against v3.3.5 succeeds.
- The operator chart renders with the cozystack-registry image override.
- The image is built (freshworks-oss v3.3.5 source + our patch,
multi-arch amd64/arm64), pushed to
`ghcr.io/cozystack/cozystack/redis-operator`, and pinned by digest in
`values.yaml`.
- `hack/promote-retag_test.bats` and `hack/nightly-mirror_test.bats`
pass with the pinned digest.

### Note for maintainers

`packages/system/redis-operator` is not part of the automated `make
build`; its image is built out-of-band and the digest is pinned in
`values.yaml`. That image is already built and pinned in this PR; re-run
`make -C packages/system/redis-operator image` only if you bump the
operator version.

### Follow-up

The cozystack/redis-operator fork carries cert-manager TLS work
(cozystack/cozystack#2729) that freshworks-oss does not have yet; the
plan is to upstream it to freshworks-oss so it lands in the shared
maintained fork rather than a cozystack-only one.

### Release note

```release-note
feat(redis-operator): migrate from the archived spotahome/redis-operator to the actively-maintained freshworks-oss/redis-operator fork (same RedisFailover CRD, adds native spec.engine: Redis|Valkey)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added support for injecting additional environment variables into the
Redis Operator (including a default operator group id).
  * Added default configurable labels for the Redis Operator.
* **Bug Fixes**
* Preserves and merges existing Service labels and annotations during
Redis Operator updates.
* **Chores**
* Updated the Redis Operator and Helm chart to the Freshworks-maintained
3.3.5 release, aligned chart/image sources, and simplified Helm chart
retrieval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-20 13:11:41 +03:00
Aleksei Sviridkin 7ec5dff93a test(e2e): add a runner-layer fixed-work canary around the node-join wait 
The counters the node-join failure path collects are read in whatever
unit their source publishes: time in the CPU rows, events in the KVM
exits, instantaneous values and running maxima in the files beside
those. Not one of them carries a unit of work of its own, so a machine
that spent every tick and got a fraction of the work done reads healthy
in all of them.

Add a collector that carries its own unit of work. It runs two fixed
amounts of work at the runner layer and times each: a loop over a
handful of integer-valued scalars, which pressure on the shared cache or
the memory controller moves little, and a store stream through blocks
larger than the last-level cache one core can allocate into, which is
dominated by it. Fixed work gives each reading a scale of its own, so a
red run can be read with no green one beside it, and the capture states
the expected range and where the estimate comes from.

Sampled where the three counter pairs already are, before the wait on
every run and after it on both paths, but outside their brackets: it
occupies a core rather than reading a file, and a burn inside one of
those intervals would land in the interval each pair divides by:
runner-kernel CPU time charged to the join window, and for the two
counting the guests, a core taken away from what they count.

Two arms, each under a twenty-second ceiling with the five-second kill
grace behind it, so fifty seconds at worst against the 420s diagnostics
budget. The budget guard derives that from the ceiling the source
declares, and the phase-warning guard names the canary among the
collectors that keep running when timeout is missing. An exit shaped
like the ceiling is attributed to it only when the whole bound was on
the clock: an outside kill produces the same status, and read by status
alone it would publish exactly the finding this collector exists to
detect.

A reading below the range the legend states, and an arm the ceiling
stopped before it could produce one, reach the job log from all three
reporting sites, each naming where it was taken, the way a missing
reading does from the two outside the diagnostics block. On the passing
path the report that holds the figure is the artifact nobody downloads.
Each floor is printed in the legend it comes from and sits where a
bounded arm crosses it before the ceiling stops the arm, because a floor
the ceiling reaches first never fires: the arm is cut off before its
rate can fall that far, and every alert then arrives as a bound rather
than as the slowdown the floor was meant to name.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-20 11:22:23 +03:00
IvanHunters cd72d81bb4 Merge branch 'main' into feat/kubernetes-nodes-phase2b 
Reconcile the phase-2b split with main's configurable worker-health
feature (MHC condition timeouts, maxNodeProvisionTime, podCpuLimit/
podCpuRequest), which landed on both charts after this branch forked:

- kubernetes (parent): pool rendering stays removed; main's parent
  worker-pool additions do not apply post-split.
- kubernetes-nodes (pool): absorb the feature; keep the directional
  parent-version skew guard and the immutable cluster/storageClass rules.
  KubevirtMachineTemplate content hashes are unchanged (render snapshot).
- migration 54: carry podCpuLimit/podCpuRequest (KMT-hash inputs) and the
  cluster-wide readyUnknownTimeout/readyFalseTimeout/maxNodeProvisionTime
  into adopted pools, so adoption neither rolls nor resets a tuned pool.
  Covered by a new adopt-values bats case.
- drop the parent chart's obsolete MHC-timeout suites; the
  kubernetes-nodes twins cover the feature.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-19 13:14:11 +03:00
Andrei Kvapil caf9511365 fix(platform): CreateReplace the redis-operator CRDs on upgrade 
helm-controller skips CRD upgrades by default, so an upgraded cluster would
keep the old spotahome schema and prune the fork's spec.engine field on
admission. The schema diff is additive, so CreateReplace is safe. Also refresh
the harbor e2e wait comment for the freshworks operator.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-19 10:15:58 +02:00
Aleksei Sviridkin b1b8cbecc2 test(e2e): price the layers above the sandbox on the node-join window 
Every CPU reading on this failure path has a subject inside the sandbox:
a cgroup, a tenant worker's threads, a sandbox node's own /proc/stat.
Each of them can say a guest was given its ticks while getting no work
done, and none can say what the layers above spent. That is where this
suite's own tax is paid: the sandbox nodes' guest time is charged to the
runner kernel as user time, and whatever the runner VM loses to the
machine hosting it is charged to nobody the sandbox can see.

Two readings close that. /proc/stat is not namespaced, so reading it from
the sandbox container reads the runner VM's kernel -- the only place
steal on that layer appears, which is what separates a slow suite from a
machine sharing a core with somebody else. The sandbox VMs are QEMU
processes in this container's own PID namespace, so their threads are a
local read too, and they say which node spent the kernel's user time and
whether the three spent it evenly.

Both are taken at the same two points as the KVM counters and for the
same reason: the values are cumulative, so a rate needs a pair, and the
interval worth measuring is the window the guest is failing to boot in.
The thread reading runs the probe the worker capture already uses rather
than a copy of it, since a second copy would drift exactly at the
failure paths.

Each carries the layer it describes in its own legend, because three
kernels in this report publish a /proc/stat and two captures read QEMU
threads, one layer apart in each direction -- and a reader who takes one
for the other compares a hypervisor against its own guest.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-18 07:25:49 +03:00
Aleksei Sviridkin 0a1d2673d0 test(e2e): capture worker block IO and volume state on node-join failure 
A worker that burns its whole vCPU while making no progress is either
computing without result or waiting on blocks, and every counter this path
already collects is blind to the second: the CPU families say what the group
was allowed and what it got, the byte counters say what crossed the network,
and neither counts a read. Read the Pod's block IO counters from the same
kubelet endpoint the CPU ones come from, once rather than twice, because they
accumulate from the container starting and a worker container starts with the
guest it carries.

Read the state of the replicated volumes behind the worker disks as well. A
replica that is SyncTarget, Inconsistent or Diskless serves the guest's system
disk from a peer over the network rather than locally, and the report collects
the same tables only after the suite, by which time cleanup has deleted the
volumes a red run would need them for.

The guest's own /proc/diskstats would answer this from inside and cannot be
reached: the credential minted for the guest carries the os:reader role, which
is not allowed to read file contents, and Talos publishes no resource carrying
disk statistics.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-17 14:22:09 +03:00
Aleksei Sviridkin 827f4e7e10 test(e2e): measure the sandbox kernel KVM exit counters across the node-join wait 
Every CPU reading on this failure path describes a subject inside the
sandbox -- a cgroup, a QEMU thread, a node's own /proc/stat -- and all of
them can report a guest that was handed its ticks while the guest gets no
work done. None of them can say what a tick cost. That price is paid one
layer down: a tenant worker is a guest of a sandbox node, which is itself
a guest of the runner, and the runner's kernel absorbs the exits, nested
page faults and TLB flushes the arrangement provokes.

Read those counters from the debugfs that kernel publishes them in. The
suite runs in a container sharing it, so this is a local read rather than
a walk over nodes, which is what makes it affordable at all: the phase
budget has no room left for a further per-node read. debugfs is mounted
in a mount namespace belonging to the call, so nothing is left behind for
the twenty minutes of suite that follow it.

The pair is taken across the node-join wait rather than inside the
diagnostics block: once before the wait on every run, once after it on
the failing path and on the passing one. The counters are running
totals -- per-VM since that VM was created, kernel-wide summed over the
VMs alive at the read -- so the interval a difference divides by becomes
the window the guest was failing to boot in, and a passing run leaves a
baseline to read the failing ones against.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-17 14:18:21 +03:00
IvanHunters 5297df5cf3 Merge remote-tracking branch 'origin/main' into pr-3315-lgtm 
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>

# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go
#	hack/e2e-chainsaw/_lib/run-kubernetes.sh
#	packages/apps/kubernetes-nodes/tests/render-parity.sh
#	packages/apps/kubernetes/README.md
#	packages/apps/kubernetes/templates/cluster.yaml
#	packages/apps/kubernetes/values.schema.json
#	packages/apps/kubernetes/values.yaml
#	packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
 2026-08-17 11:25:29 +03:00
Aleksei Sviridkin 73b8afc256 test(e2e): run tenant workers on one vCPU with CPU headroom above it 
Under a two-vCPU worker both vCPU threads burn 1.4 to 1.8 cores between
them while the guest kernel never gets past SMP bringup, and raising the
CFS ceiling from two cores to three changed neither the burn nor the
progress, so the spin is between the two vCPUs, not against the quota.
A single-vCPU guest has no sibling to spin on. The ceiling of two keeps
the QEMU emulator and IO threads from eating into the one core the
guest computes with, and the request keeps scheduling where the workers
always were.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-17 03:09:04 +03:00
Aleksei Sviridkin b8964c8233 test(e2e): give tenant workers CPU headroom with a request to match 
Restores the ceiling the kubernetes suites briefly carried, now that a
node group can declare the CPU request that goes with it. The limit on
its own was defaulted into the request by KubeVirt, so every worker
asked the scheduler for the whole ceiling and sat Pending on
Insufficient cpu before a guest existed. The request is set to what
KubeVirt derived for these workers before, which leaves scheduling
where it was and leaves the ceiling as the only variable the run
changes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-16 21:59:19 +03:00
Aleksei Sviridkin 80e2753bf4 fix(e2e): drop the worker CPU limit until it can carry its own request 
A container that sets a CPU limit and no request is given a request equal
to the limit, so each tenant worker asked the scheduler for its whole
three-CPU ceiling and every kubernetes suite worker sat Pending on
Insufficient cpu across all three sandbox nodes, before any guest booted.

The headroom experiment comes back when the chart can pair the limit
with an explicit request.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-16 21:01:30 +03:00
Aleksei Sviridkin c2c50b4119 test(e2e): give tenant workers CPU headroom above their vCPU count 
Workers that fail to join have been measured burning their vCPU threads
flat out under a ceiling equal to their vCPU count while the guest
kernel recorded no progress at all, with the emulator, IO, vhost and
vnc threads flat at zero. That is the shape of a vCPU spinning on a
lock whose holder is the other vCPU, preempted out of the quota the two
share, and it predicts that a ceiling with room above the vCPU count
lets the preempted one back in.

What the next runs answer is whether the join failure rate moves. If it
does not, the lever is elsewhere and the value goes back.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-16 17:06:16 +03:00
Aleksei Sviridkin cf2d9c6d13 test(e2e): split the tenant worker CPU time across QEMU threads 
The node-join diagnostics phase samples the tenant worker's CPU
counters twice on failure, but it reads them from the compute
container's cgroup, and that cgroup holds every thread QEMU runs:
the guest's vCPUs, QEMU's main thread, and its IO and worker
threads. A container burning most of its quota while the guest
reports no progress is two incompatible findings wearing one
number, and no container-level counter separates them.

Add a collector that execs into each Running worker's compute
container, finds the QEMU process and copies every
/proc/<pid>/task/<tid>/stat line into the artifact verbatim, with a
legend beside them. Nothing is computed here. The legend names
where utime and stime sit and why counting fields over whitespace
reads the wrong column for exactly the threads the split is about:
a KVM vCPU thread is named "CPU 0/KVM", that space is inside the
comm parentheses, and every field after it shifts by one.

It runs inside the sampling loop the two existing subjects already
share, so it costs no second wait: both readings ride the interval
already being paid, and the rate comes from the two files plus the
stamps each one carries.

Every way the read can come up short is a named outcome rather than
an empty directory, and the legend and the pairing instruction are
withheld from each of them: telling a reader to subtract two files
asserts that both hold a whole reading. The probe shells out to
nothing but getconf, whose absence it names, because a probe that
needed cat would report a container missing coreutils as a
container whose QEMU had exited, and it counts what it managed to
read so that having read no process at all stays separable from
having read them all without finding QEMU. Its stderr is suppressed
on the walk rather than on either read: redirections are applied
left to right, so a 2>/dev/null after an input redirect is not in
place yet when opening the input fails, and a thread that exits
between the glob and the read would otherwise put a shell
complaint in the artifact beside a healthy capture. It also reports
a QEMU that was named and then gone before its task files could be
opened, because a process heading alone is enough to make the
capture non-empty and would otherwise carry a legend describing
columns the file does not have.

The Pod listing the cAdvisor captures already share now takes its
whole query as a parameter, so this collector reuses that read
rather than carrying a third copy of it, and asks for the Running
Pods only: a launcher left over from an earlier attempt costs the
sibling caller nothing and would cost this one a bounded exec and a
slot under its cap. The guard that holds the sampling group against
the room the tenant snapshot needs now counts every capped walk in
the loop rather than the two spelled the same way, so the new
subject is inside the arithmetic rather than beside it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-16 05:17:16 +03:00
Aleksei Sviridkin 959e90c9ab test(e2e): time the guest console capture and baseline it on a green run 
A tenant worker's guest console is stamped in seconds since that guest's
own kernel started, so a log ending mid-boot says how far the guest got
and nothing about when. Whether it ends there because the guest started
late or because the guest stopped printing decides which failure is being
looked at, and neither instant was recorded: the read's own instant was
nowhere in the file, surviving only as the artifact's mtime, which is
metadata rather than content and so says nothing to anyone reading the
log; and the guest's start only inside a describe dump that a capture
with no silent console never triggers. Across three collected runs every
capture exited zero and every log began at the first kernel line, so the
reads were complete at both ends and the logs end where the guests
stopped -- and which of the two happened could not be read off any of
them.

Each console log now carries the instant its own read started and
returned, and CAPTURE-TIMELINE.txt beside it carries each Pod's phase,
its start time and the start of its compute and console containers, from
one bounded read taken once for the whole selector. Those subtract to how
long a guest had been alive when it was read, which is what the log's
last kernel stamp is then compared against. The console container is
asked for in both status lists, since KubeVirt runs it as an init
container and reading only one of them would report a container that
started fine as one that never ran. A read that failed, a read cut off
part way and a read that answered with nothing are recorded as
themselves rather than left as a missing file, and the note names the
error log only when there is one.

The same capture also runs on the passing path, before the tenant is torn
down and the Pod holding the stream goes with it. Every console this tree
has collected came from a failure, so what a healthy boot looks like has
never been measured, while "slower than usual" is the claim the failure
keeps resting on. That caller asks for the pool's minimum rather than its
maximum, since two workers booting is the same measurement as ten, and it
warns in the job log rather than failing a suite that has already proved
everything it exists to prove. Both paths write the same directory, so
the capture also records which of them produced it.

That capture costs real wall clock at the tail of an operation with a
fixed ceiling, which is a way for a run that passed to be killed while
collecting a debugging aid. So it is declined out loud when the ceiling
no longer has room for it, against a reserve derived from its own cost
and held in step with the timeout both suites give the operation.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-15 23:33:33 +03:00
Aleksei Sviridkin 2b3c163d15 fix(e2e): size tenant workers by resources, not by u1.large 
Sizing the workers with u1.large doubled their memory along with the
vCPUs, and four 8Gi workers do not fit on the sandbox beside the rest of
the suite: the second worker of each cluster stayed Pending with
Insufficient memory on all three nodes, so the node group could never
reach two Ready nodes and node-join failed for a reason that has nothing
to do with the CPU ceiling it was meant to test.

Setting resources.cpu and resources.memory keeps the two vCPUs at the
memory the workers had before. The chart drops the instancetype from the
VM when a group sets both, and the values schema still requires the
field, so it stays in the values as the sizing fallback it now is.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-15 18:34:13 +03:00
Aleksei Sviridkin 3b49be07fa test(e2e): give tenant workers two vCPUs 
The tenant worker boots Talos and unpacks the kubelet image on a single
vCPU whose CFS quota also covers the QEMU emulator and IO threads, and
measurements on both quiet and loaded runs show the compute container
pinned at 93-98% of that one-core ceiling for the whole node-Ready
budget. The join deadline is then decided entirely by how much CPU work
the boot happens to need, and under parallel e2e load it does not fit.

Doubling the worker to u1.large raises the ceiling instead of the
weight, which measurements showed changes nothing since the workers
never lose a scheduling fight.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-15 15:08:31 +03:00
Aleksei Sviridkin 527d4ea00f test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it 
Two instruments for the node-join failure the kubernetes-* suites keep
hitting. Neither changes what the suites assert.

The worker CPU capture now reads container_cpu_usage_seconds_total
beside the CFS counters, and reads them twice a fixed interval apart
instead of once. The throttled counters say a container met its ceiling
and cannot say how much CPU it got, which are the two candidate
explanations for a stalled worker: a guest held at a one-core ceiling,
and a guest whose vCPU thread is rarely scheduled at all. Both leave
throttled periods behind. Every counter involved accumulates from
container start, so a single reading is an average over the whole uptime
and no rate at all, and the profile under investigation is a guest that
freezes for tens of seconds and then runs flat out: the average comes
out low, the instantaneous figure comes out at the cap, and only the
second separates it from a guest that is simply capped.

The sandbox nodes are a new subject. The worker captures describe a
cgroup, so a guest that is runnable and never scheduled looks from there
like a guest that asked for nothing. Whether the sandbox node itself was
given its turn is recorded only in the steal column of its own
/proc/stat: no kubelet endpoint carries it, and crust-gather cannot
reach it because PodSecurity refuses its node-shell Pod here, so the
read goes over the Talos API, which is already how the report reaches
those nodes for dmesg. The rows are captured verbatim with their column
legend beside them rather than reduced to a percentage, because
/proc/stat has no header, steal and guest sit next to each other, and
guest is large on a node running VMs and already counted inside user.

Both subjects are read on either side of one shared wait, so the pair
pays it once. The wait is not the interval either subject's counters
span, though: each pass walks its own nodes and the other subject's pass
falls between a subject's two readings, and on the run this exists for
those passes are the slow part. Each capture therefore stamps both ends
of its read, so a rate comes from subtracting stamps rather than from
assuming the knob, and the width of those brackets is the uncertainty.
That wait is a measurement interval rather than a stand-in for a
condition nobody wrote a wait for, so the e2e conventions carve it out
by name instead of marking it as a fixed timeout to be replaced.

The pair is gated below the serial console rather than above it. Its
ceiling is sixteen bounded reads plus the interval, which is most of the
phase budget, and admission gates only when a collector may start, so
above the console a run that hit those bounds would leave the console
never started rather than merely cut short. The console is the only
capture that survives a worker which never reached apid, which is the
shape this failure usually takes. That overrun is inside the term the
budget derivation already allows for one late admission, and a guard
holds the pair's ceiling against it.

No collector here may leave an empty directory, since an empty directory
reads as a healthy node: a missing binary, a missing talosconfig, a
listing that never answered, a node with no address and a read that was
cut off each land as a named note or a COLLECTION-FAILED.txt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-15 10:19:19 +03:00
Aleksei Sviridkin 3dd142a718 fix(e2e): capture both ends of the worker byte path on node-join failure (#3804) 
## What this PR does

A tenant worker that misses the node-Ready deadline while its kubelet
image pull crawls leaves two mechanisms behind one symptom, and the
failure artifact could not tell them apart: either the link is slow, or
the pull keeps restarting and discarding what it fetched. From inside
the guest both are a progress line that barely moves. This adds the
host-side counters that separate them, and the guest-side reads that
explain a kubelet which never got as far as pulling at all.

**Host side.** A per-worker network capture reads the eight
`container_network_*` families from the same kubelet cAdvisor endpoint
the CPU throttling capture already uses. cAdvisor builds them from the
interface counters inside the Pod's own network namespace and publishes
one row per interface, so what arrived at the worker is counted there in
full, including the bytes a restart threw away, set against the progress
the guest reported, that separates a pull which kept restarting from a
slow link.

Which row answers is not obvious, and getting it wrong inverts the
reading, so the collector states it rather than leaving it to the
reader. A bridge-bound worker presents four interfaces in its Pod netns:
the one still named `eth0` is a dummy that holds the Pod IP so the
kubelet's check passes and carries no traffic, so it reports zero, and
it is the row a reader reaches for first, by name, where zero reads as a
worker that received nothing. The number that answers is the renamed
NIC's receive; the bridge counts the same bytes again, and the tap's
counters are named from the host's end. The filter deliberately carries
no interface predicate, so all four reach the artifact and can be told
apart.

Two further properties of the family fail by capturing nothing rather
than loudly, so both are stated where the filter is: the dropped-packet
series are spelled `packets_dropped` and there is no `*_drops_total`,
and every row carries an empty container label because cAdvisor
publishes them on the Pod sandbox, which means a filter borrowed from
the CPU counters beside it matches nothing at all.

The fixture's row shape and values are real kubelet output with the
identifying labels rewritten. Its four interface names are derived
rather than observed: no virt-launcher Pod was available to sample, so
they are read from the name generators in the KubeVirt version this repo
ships, and the counters attached to them illustrate direction rather
than magnitude.

**Guest side.** The Talos capture gains the service state machine and
the link table. Talos creates a service's log buffer when its runner
opens the log writer, as it starts the process, so a kubelet still
waiting on its volumes, on time sync, or on the container runtime
reporting healthy never reached its runner and has no log at all, and
`talosctl logs kubelet` then fails naming a log that was never
registered, which is true about the log and silent about the service.
`talosctl services` answers with the state and the condition being
waited on, in one call for every service, so the runtime a kubelet is
blocked behind arrives in the same file. The link table is read as YAML
because MTU lives in the `LinkStatus` spec and not among the resource's
printed columns, and it is the value that decides whether a path-MTU
reading of a stalled transfer survives.

Both new reads are made with the short-lived `os:reader` credential the
capture already mints; neither needs a wider role.

**A worker that answered nothing now says so once, for the worker.**
Each read already carried its own error and exit code, which is accurate
per file and adds up to a directory indistinguishable from a worker that
simply had little to report.

### What this does not do

This collects evidence; it fixes nothing and is not expected to change
any suite's outcome. The comparison it enables is not symmetric either,
and the source says so: the host side reports bytes on an interface,
while the guest side reports an image pull's own progress, which is
unpacking rather than arrival. They bound each other; they are not two
measurements of one quantity.

### Cost

The diagnostics phase budget drops from 8m to 7m. That is forced rather
than chosen: the inequality that keeps the tenant crust-gather snapshot
reachable had ten seconds of slack, and the two added guest reads cost
sixty across the minimum worker pool. A shorter budget declines more,
and what it declines first is what is gated last: the mirror capture,
whose subject is partly recoverable elsewhere, and the image-cache
re-probe, which has no substitute and was already first to go. The new
network capture also spends a listing and one read per node ahead of the
guest captures, so those reach their gate with less headroom than
before. All of it is priced where the number lives, as a ceiling the
read bounds impose rather than a duration measured on a live cluster,
and a collector the budget declines says so in the log rather than going
quiet.

The remaining slack is ten seconds, which is worth knowing before the
next collector is added rather than after: it buys no bounded read at
any bound this file uses, so adding one means re-deriving the budget
again.

### Testing

`hack/run-kubernetes-network-counters_test.bats` is new (15 tests). The
node-join, Talos-diagnostics and CPU-throttle suites gain coverage for
the phase ordering, the per-worker marker boundary, the spend-order
comments in both Chainsaw suites, and the enumeration of collectors that
survive a missing `timeout`. Every claim the two Chainsaw comment blocks
make about spend order is derived from the source rather than restated,
so a collector added without a line there fails.

The two cAdvisor captures now run one shared body. They previously
carried near-identical copies, and the copies cost more than duplication
normally does: the block-level bound audit pins each collector by
something only it produces, and two collectors issuing identical reads
left neither with anything of its own, so removing one from the audit
went unnoticed. Each is now pinned by the report directory it writes.

Verified with `hack/cozytest.sh` across eleven suites, checking that the
number of tests executed matches the number declared, because an exit
code answers "did anything fail", never "did anything run". Each
behavioural change was confirmed by mutating the source and observing
the intended test fail.

### Screenshots

No UI changes.

### Downstream repositories

Walked the trigger map file by file against the diff. All seven changed
files live under `hack/`, and the change adds one file and modifies six;
no repository file is moved, renamed or deleted. The `ccp` entry is the
only one that keys on `hack/` at all, and it keys on `hack/package.mk`,
`hack/common-envs.mk` and `hack/update-crd.sh` as anchors for `make
generate`; none of those is touched, and no make target changes
behaviour. Nothing here is a package, a chart value, an
`ApplicationDefinition`, a CRD, a release asset, a namespace, a variant,
or a node prerequisite, so no other entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added worker network-counter diagnostics with interface-level and
cumulative traffic data.
* Expanded Talos diagnostics with service-state and link-state
information.
* **Bug Fixes**
* Improved reporting for partial captures, timeouts, unavailable data,
and zero-traffic workers.
* Clarified failure reporting and fallback messages when timeout
controls are unavailable.
* **Documentation**
* Updated diagnostic timing and collector documentation for the
seven-minute budget and new metrics.
* **Tests**
* Added comprehensive coverage for network counters, filtering,
truncation, and expanded Talos diagnostic scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-15 03:44:05 +03:00
Aleksei Sviridkin e441edc2aa test(e2e): stop the mongodb backup suite before the restore leg 
A to-copy restore replays the source cluster's system users into the
target, so the target's own generated credentials stop authenticating
and the harness cannot read the sentinel back out of the restored copy.
The suite has never passed for that reason, holding the in-tree e2e run
red on every branch that selects it.

Drive run-all.sh with SKIP_RESTORE=1 so the suite still proves the
backup half end to end, and pin the skip by asserting the skip branch's
own log line: exit zero alone is also what a completed round-trip
returns, so without that assertion the step could not tell a skipped
flow from a full one, nor notice SKIP_RESTORE failing to reach the
harness.

The to-copy manifest and the example README both stated that CI covers
the restore, which stops being true here, so both now say where
coverage ends. The asserted log line is annotated at its source, since
rewording it would red the suite with a bare match failure.

Coverage returns by dropping SKIP_RESTORE together with the assertion
on that line, once the credential handling behind the restore is
fixed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-14 21:18:09 +03:00
Aleksei Sviridkin 61edf33ef4 fix(e2e): capture both ends of the worker byte path on node-join failure 
A tenant worker that misses the node-Ready deadline while its kubelet
image pull crawls leaves two mechanisms behind one symptom, and the
artifact could not tell them apart. Either the link is slow, or the pull
keeps restarting and discarding what it fetched; from inside the guest
both are a progress line that barely moves.

Per-worker network counters, read from the kubelet's cAdvisor endpoint
the CPU throttling capture already uses, count every byte that crossed
into the Pod including the ones a restart threw away, so the gap
between that and the progress the guest reported separates the two.

Which row carries it is not obvious and getting it wrong inverts the
reading, so the collector states it. cAdvisor publishes one row per
interface in the Pod netns, and a bridge-bound worker has four: the one
still named eth0 is a dummy holding the Pod IP, carries no traffic, and
reports zero -- the row a reader reaches for first, where zero reads as
a worker that received nothing. The renamed NIC's receive is the number
that answers; the bridge counts the same bytes again and the tap is
named from the host's end. No interface predicate is applied, so all
four reach the artifact and can be told apart.

Two further properties of the family fail by capturing nothing rather
than loudly: the dropped-packet series are spelled packets_dropped, and
every row carries an empty container label because cAdvisor publishes
them on the Pod sandbox, so a filter borrowed from the CPU counters
beside it matches nothing at all. Both are stated where the filter is.
The fixture's rows are real kubelet output relabelled; its interface
names are read from the KubeVirt version this repo ships rather than
sampled, since no virt-launcher Pod was available.

The guest side gains the service state machine and the link table. Talos
creates a service's log buffer when its runner opens the log writer, as
it starts the process, so a kubelet still waiting on its volumes, on
time sync, or on the container runtime reporting healthy never reached
its runner, has no log at all, and the capture recorded an rpc error
naming a log that was never registered: true about the log, silent about
the service, while the state and the awaited condition were the
finding. The link table is read as yaml because MTU lives in the spec
rather than among the printed columns, and it is the value that decides
whether a path-MTU reading of a stalled transfer survives. Both carry a
tighter bound than the two reads beside them, being single tabular RPCs
rather than streams, and the walk they sit in is the term the phase
budget sizes itself against.

A worker whose reads all failed now says so once for the worker. Each
log already carried its own error and exit code, which is accurate per
file and adds up to a directory indistinguishable from a worker that had
little to report.

The phase budget drops from 8m to 7m to pay for the two added guest
reads, because the inequality keeping the tenant snapshot reachable had
ten seconds of slack. A shorter budget declines more, and what it
declines first is what is gated last: the mirror capture, whose subject
is partly recoverable from its own request counts and the host snapshot,
and the image-cache re-probe, which has no substitute and was already
the first to go.

The second cost lands on the guest captures, which are gated after the
new counters and reach their gate with both the shortened budget and a
listing plus a read per node already spent -- up to 100s at the read
bound and the three-node cap, a ceiling those numbers impose rather than
a duration measured on a live cluster. Sharing that read would mean the
two captures becoming one walk, which is left out for scope rather than
merit: the phase gate is a deadline rather than a per-collector
allowance, so one walk admitted at the first of the two gates would cost
less and collect more. The cost is stated rather than netted away, and
the source says which way the argument runs.

Both captures run one shared body, which also settles what the two
copies had already broken once: the audit that catches an unbounded read
pins each collector by something only it produces, and two collectors
issuing byte-identical reads left neither with anything of its own, so
stubbing one out of the audit went unnoticed. It is pinned by the report
directory each writes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-14 14:34:37 +03:00
Aleksei Sviridkin 829edfadb3 test(e2e): warm the ghcr.io mirror before the tenant suites need it 
The in-sandbox pull-through cache fetches a blob only when a client
asks for it, so the first tenant worker's kubelet pull is a live fetch
from ghcr.io. That leg is what misses the node-Ready budget: measured
at 40-165 KB/s for a 61 MiB image, with two blob streams dying
mid-transfer after 7-8 MiB on an upstream HTTP/2 reset. Talos holds
the kubelet service in PreFunc until the pull completes, so the node
never registers, and the tenant cilium HelmRelease then fails for want
of a schedulable node rather than for any fault of its own.

Fill the cache from a Job started in the same install step that
deploys the mirror, tens of minutes before the kubernetes suites run.
The Job waits for the mirror to serve, then asks it for each kubelet
tag's index, walks the manifests it gets back, and discards the blob
bodies; distribution populates its blobstore on the way past.

It warms one platform per tag -- the one the workers run -- picked out of
the index by record, which keeps the walk a grep. Walking both platforms
was a true addition to the same throttled leg: arm64 is bytes nothing in
the sandbox pulls, and the digest ordering put it first for both tags, so
on the slow runs this exists for the deadline expired with the second tag
still cold.

It warms the two minors the suites select, not the whole version map.
The map would be ~300 MiB at the one-platform walk this ships (~61 MiB
per tag), three fifths of it for minors no suite can resolve, spent on
the same throttled egress the rest of the install is using. The tag
list still comes from the chart's map rather than a copy, ordered the
way the suites resolve it, so the set is theirs without restating
their expression.

The readiness wait is a wall clock, not a per-attempt count. Cilium
answers a Service with no ready endpoints by rejecting, so a probe
costs nothing when the mirror is absent and the full timeout when it
is merely slow; a constant was wrong in both directions across two
attempts at picking one.

Tags are refused by character class before the pattern is applied, so a
value cannot pass by having a valid first line: grep matches per line
and the pattern's anchor is end-of-line, not end-of-input.

The Job exits non-zero on any miss, so backoffLimit is a real bound
and a warm-up that reached nothing cannot report success. It names its
own requests, because the node-join diagnostic counts kubelet GETs in
the access log to tell a worker that pulled through the mirror from
one that fell back to public ghcr.io.

Naming them is only half of that. The same diagnostic now reports the
Job's status and result and the Job outlives the suites, because
excluding the warm-up without reporting it would leave a red run
unable to distinguish a warm cache from a warm-up that never started.
That counter also moved onto the access-line shape: registry:2.8.3
writes three lines carrying the path per request, so matching the path
counted one pull as three. It reads the repository from the same
variable the warm-up does, so a rename cannot leave the diagnostic
counting zero while the workers pull under the new name. GET and
HEAD both count: containerd resolves a manifest with a HEAD first,
so a GET-anchored count reports a pull that died before its first
GET as no request at all.

The failure path also prints how long the mirror took to answer the
worker. A warm cache and a cold one are otherwise indistinguishable in
the artifacts -- the Job reports success either way, because the proxy
stores the blob in the background and a completed GET is a different
event from a stored blob -- and server-side duration separates them by
orders of magnitude. It filters the log in the shell that read it: a
single argv element is capped at 131072 bytes on Linux regardless of
ARG_MAX, and probe traffic carries this log past that within about
twenty minutes of mirror uptime, so handing it to a helper would empty
the section on every run worth reading. Nothing is printed when the log
could not be read at all, because a heading with no lines under it
reads as a finding about the workers.

Measured against a local pull-through registry using the script the
builder emits: 32m47s cold, 1.1s warm for the same tags, and the
blobs it fetches are the same digests the failing runs stalled on.

A warm-up that fails, times out or never starts does not fail the
step, and the suites behave as they did before it existed. It is not
free, though: it pulls across the same throttled egress the install
is using, which is why the warm set is two tags and not five.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-14 11:08:49 +03:00
Aleksei Sviridkin 9dc49de1c9 fix(apps): make the post-delete cleanup hooks report failure (#3593) 
## What this PR does

The qdrant and harbor post-delete cleanup hooks could not fail. Every
delete was guarded by `|| echo "WARNING: ..." >&2`, neither script set
`-e`, and both ended on an unconditional completion message, so the
container exited 0 on every path. Helm read that as a successful hook,
`helm.sh/hook-delete-policy: hook-succeeded` removed the Job together
with the pod that held the warning, and an uninstall that reclaimed
nothing looked exactly like one that had nothing to reclaim. Written up
in #3566.

Both hooks now count failures and set the exit code at the end. `set -e`
would abort at the first failed delete and skip the rest of the sweep,
which is the opposite of what is wanted: every remaining object should
still be attempted and the outcome reported once. Already-gone objects
stay a success, since every delete keeps `--ignore-not-found`. Each
failure message names the object it left behind. `packages/apps/mariadb`
already uses this shape.

Both hooks also consumed `kubectl get` output directly: harbor by
iterating `$(kubectl get ...)`, qdrant by piping it into `grep`, whose
status the pipeline then reports. A failed listing arrived as an empty
list, and the sweep reported success without having looked. Each listing
is now captured before it is used.

#3566 attributes that shape to harbor's two loops alone. qdrant has it
too: `pvcs=$(kubectl get pvc ... | grep -E ... || true)` reports grep's
status, so a `kubectl get` that failed left `$pvcs` empty and the hook
announced "no data PVCs found; nothing to delete". Both instances are
fixed here.

### The cost

A hook that exits non-zero keeps its Job and its pods, because
`hook-succeeded` does not fire for it. That is the point, the pods hold
the log, but it means a failed uninstall now leaves objects behind where
it previously left none. Nothing collects them on its own:
`before-hook-creation` only fires when the hook runs again, and a
`post-delete` hook runs only on an uninstall, so they stay until a
release of that name is installed and uninstalled again, or until the
namespace goes.

The release itself is not stranded. Helm appends the post-delete hook
error to its error list instead of returning on it, marks the release
uninstalled, purges it from storage anyway, and only then reports
`uninstallation completed with N error(s)`. The failure surfaces once
and the app still goes away.

Both Jobs get one retry instead of the Kubernetes default of six, and
instead of none. Every delete is idempotent, so a second attempt
re-issues an API call rather than re-running any logic, and one short
backoff separates a transient apiserver error from a real one. These
failures become visible for the first time with this change, and a
signal that fires on noise stops being read, including on the day it is
right. Six is the far end of the same mistake: ten minutes of backoff
across seven pods outlasts the hook timeout, so Helm reports that
timeout instead of the hook's own error.

`packages/apps/clickhouse` and `packages/apps/mariadb` already exit
non-zero from their cleanup hooks and set no retry limit, so a real
failure in either spends the full backoff and reaches the operator as a
timeout rather than as the error the hook wrote. Adding the exit code
without the cap reproduces that.

One limitation is deliberate. harbor's ACME sweep cannot tell "this kind
is not served" from "I could not look". If cert-manager is torn down
before the release, `kubectl get challenge` fails and the uninstall is
reported failed with nothing to sweep. cert-manager is a hard dependency
of the bundle harbor ships in, so the window is narrow, and a
CRD-existence probe would add a second silent-skip path of its own: it
would swallow a discovery failure the way this change stops swallowing a
listing failure.

### Tests

`hack/app-cleanup-hooks_test.bats` renders each hook's script out of its
chart and runs it against a fake `kubectl`, because the property under
test is the container's exit code and matching against manifest text
cannot establish it. There are eight failure paths across the two hooks,
six in harbor and two in qdrant, and each has a case that goes red when
that one path alone is reverted to a swallowed warning. Also covered: a
clean delete, and an object that is already gone in both its shapes, an
empty listing and a `NotFound` on the delete itself.

The fake `kubectl` models `--ignore-not-found` instead of exiting 0
unconditionally, so dropping that flag from a hook turns the
already-gone cases red. Message assertions are anchored on the hook's
own `ERROR: ` prefix, because the fake names the object in its own
diagnostic too, and an unanchored match is satisfied by the fixture.

Two manifest fields are pinned in each chart's existing helm-unittest
suite, where manifest fields belong. The retry cap is one. The other is
`helm.sh/hook-delete-policy`: adding `hook-failed` to that annotation
hands the failed Job straight back to Helm to delete, which undoes the
whole change. I checked that before adding the assert, and the reversal
left both suites fully green.

The qdrant Chainsaw suite explained its post-delete pod collector by
quoting the guard string this change removes, and concluded that Helm
always collects the pod away. That conclusion inverts here, so the note
is rewritten, along with the sentence in the harbor suite that points at
it.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file. It touches two app chart templates and adds one file
under `hack/`. No package added, renamed or removed, no
`values.schema.json`, no `values.yaml` default, no version enum, no
`ApplicationDefinition`, no namespace, no node prerequisite, and neither
`hack/package.mk` nor `hack/update-crd.sh`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(apps): the qdrant and harbor post-delete cleanup hooks now exit non-zero when a delete or a listing fails, instead of reporting success on every path. `helm uninstall` reports the failure rather than returning clean, and the hook Job and its pods stay in the namespace so the log survives to be read. Nothing collects them on its own: they remain until a release of that name is installed and uninstalled again, or until the namespace is removed.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved Harbor and Qdrant cleanup reliability by detecting listing
and deletion failures.
- Cleanup jobs now retry once and report failures with a non-zero
status.
- Cleanup continues processing remaining resources after individual
failures.
- Successful cleanup remains safe and repeatable, including when
resources are already missing.
- Harbor cleanup remains limited to resources associated with the
configured Harbor host.

- **Tests**
- Added comprehensive coverage for successful, repeated, partial, and
failed cleanup scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-14 10:55:01 +03:00
IvanHunters 09dc7f47fb feat(backups): add a native MongoDB (psmdb) backup-strategy driver (#3562) 
## What this PR does

Adds a native MongoDB (Percona Server for MongoDB / psmdb)
backup-strategy driver so `apps.cozystack.io/MongoDB` reaches feature
parity with the CNPG, MariaDB and ClickHouse native drivers — logical
backups, restore into a differently-named instance, and point-in-time
recovery — all through the unified `BackupJob` / `RestoreJob` /
`BackupClass` / `Plan` interface (closes #3545).

The driver orchestrates the operator's `PerconaServerMongoDBBackup` /
`PerconaServerMongoDBRestore` CRDs:

- **BackupJob**: creates a one-shot `PerconaServerMongoDBBackup`, polls
`status.state`, and persists the S3 destination + storage descriptor (by
reference — Secret name only, never a raw credential) onto the Cozystack
`Backup` artifact.
- **RestoreJob**: drives `PerconaServerMongoDBRestore` via
`backupSource` (read back from the source backup status, with a snapshot
fallback for a reaped operator CR) so the same artifact restores
in-place **or** into a differently-named instance. PITR is exposed as
`spec.options.recoveryTime` (RFC3339), matching the CNPG driver's option
exactly; unrecognised `spec.options` keys are surfaced as a
`UnknownRestoreOption` Warning event rather than silently dropped.

Because psmdb resolves the backup storage by name against the cluster's
`spec.backup.storages`, the driver names the chart's storage
(`s3-storage`) via the `cozy-default-mongodb` Strategy rather than
injecting it, and surfaces a clear `Ready=False` precondition when a
target has backups disabled instead of hanging until the deadline.

**Wiring**

- `api/backups/strategy/v1alpha1`: `MongoDB` strategy type + generated
CRD/deepcopy.
- `internal/backupcontroller`: `psmdbstrategy_controller` + `psmdbtypes`
/ `mongodbapp` partial-schema mirrors; BackupJob/RestoreJob dispatch,
supported-kinds, cleanup-on-delete branch, and scheme registration.
- `backupstrategy-controller`: `cozy-default` MongoDB route,
`cozy-default-mongodb` Strategy CR, and `psmdb.percona.com` RBAC
(get/list/watch, +create on backups/restores, no delete — the driver
does not own the S3 archive).
- `apps/mongodb`: a `backup.insecureSkipTLSVerify` value (psmdb's s3
storage has no CA-bundle field) so backups can target Cozystack's
self-signed in-cluster seaweedfs. Default `false`, fully
backward-compatible; regenerated schema/README/RD.

**Tests + e2e**

- Unit tests for the driver: applicationRef validation,
ensure-idempotency (backup + restore), the backups-enabled/storage
precondition gate, `backupSource` resolution (live-preferred / snapshot
fallback / no-destination), snapshot round-trip, the producer-seam
(`createMongoDBBackupArtifact`) driver-metadata, the cleanup-on-delete
no-Velero-fall-through guard, and `recoveryTime` → psmdb-UTC mapping +
unknown-option reporting.
- `examples/backups/mongodb/` + a `mongodb-2-backup-roundtrip` Chainsaw
suite that drives `run-all.sh` end-to-end (a sentinel document
round-trips through S3 into a restored copy while the source stays
untouched), per `docs/agents/e2e-testing.md`.
- `docs/operations/backup-classes.md` updated (new route, the
app-owns-the-storage model, and the `recoveryTime` PITR usage).

Verified end-to-end on a live dev cluster against the platform-default
`cozy-backups` bucket: backup via `cozy-default` reached the bucket and
the Cozystack `Backup` went `Ready`, and a to-copy `RestoreJob` restored
the sentinel into a differently-named instance with the source left
intact.

### Screenshots

N/A — no UI changes (controller, chart values, CRD, docs and tests
only).

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file by file
against the diff:

- **cozystack/website** — the only app-facing change is
`packages/apps/mongodb` (a new `backup.insecureSkipTLSVerify` value +
regenerated `README.md`). The managed-app reference page is regenerated
from the package `README.md` by the release bot for apps already in the
website `Makefile` (mongodb is one), so it refreshes automatically on
the next tag — no manual follow-up. `docs/operations/backup-classes.md`
is not a website-mirrored doc in the trigger map.
- **cozystack/terraform-provider-cozystack** — the one candidate trigger
("a field added to an app's `values.schema.json`") lands inside the
mongodb `backup` block, which the provider's
`internal/provider/mongodb_schema.go` explicitly states it does **not**
manage ("The deprecated backup block, the bootstrap restore block, and
detailed shardingConfig are not managed"). No enum, default,
`release.prefix`, Secret/Service name, or hand-typed CRD (`Plan`,
`RestoreJob`) changed; the new `strategy.backups.cozystack.io/MongoDB`
CRD is opaque to the provider. Not affected.
- All other repositories — no matching trigger (no `hack/` layout or
make-target change, no namespace rename, no `ApplicationDefinition`
semantics change, no `packages/core/platform` values, no telemetry
metric or `cozy-proxy` label rename).

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(backups): add a native MongoDB (psmdb) backup-strategy driver — logical backup to the cozy-default BackupClass, restore into a differently-named instance, and point-in-time recovery via spec.options.recoveryTime
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added MongoDB backup and restore support, including scheduled and
on-demand backups, restores to new or existing clusters, and
point-in-time recovery.
* Added configurable S3 TLS certificate verification for MongoDB
backups, defaulting to secure verification.
  * Added MongoDB support to the default backup configuration.

* **Documentation**
* Added MongoDB backup and restore examples, workflows, and operational
guidance.

* **Tests**
* Added automated coverage for MongoDB backup, restore, cleanup, and
end-to-end scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-13 11:18:46 +03:00
Aleksei Sviridkin b8a5b9ba1d test(e2e): capture worker CPU throttling counters on node-join failure 
When a tenant worker misses the node-Ready deadline, the sandbox node
it runs on is routinely at half its capacity with a clean kernel log.
In that state "the host starved the VM" and "the VM was held at its
own CFS ceiling" produce identical evidence: node-level utilisation
reports what the node used, never what the container was allowed. The
CFS counters are the only place the difference is recorded, and
nothing in the tree collected them for the tenant workers, so every
diagnosis of this failure so far had to be inferred from outside.

Read them from the kubelet's cAdvisor endpoint on the node hosting
each worker. Five series carry the whole answer and cAdvisor publishes
all of them per container, so nothing is computed here: the scheduled
and throttled period counts, the time spent throttled, and the quota
and period that form the ceiling. The throttled counters alone say a
container hit some ceiling, not which one, and a VM capped at one core
and a VM capped at eight are the same number without the quota.

Absence of the quota series is itself a reading rather than a gap.
cAdvisor emits it only when the quota is non-zero, so a container
reporting the period and no quota is one running uncapped - which is
one of the two answers this collector exists to distinguish, arriving
without being derived.

Reading the kubelet rather than the container matters beyond
convenience. A reader running inside the container shares the cgroup
it measures, so it slows down in proportion to how much the answer
matters and would need a budget sized for the pathological case; the
kubelet is outside that cgroup. It also drops any dependency on what
the container image ships and on how the host lays cgroups out.

The namespace is not a worker filter and is not used as one: it also
holds the tenant control plane, whose apiserver and etcd carry CPU
limits of their own and can be genuinely throttled. The Pod name
carries the second half of the filter, so a throttled apiserver cannot
answer under a heading that says worker.

The walk is over nodes rather than Pods, because the endpoint is per
node and one read covers every worker it hosts. It is gated on the
diagnostics phase budget, and placed first among the five gated
collectors: the budget declines whatever has not started when it runs
out, its own ceiling is four bounded reads rather than the minutes the
four after it can spend, and it is the only one whose question has no
other answer in the tree. That order is a trade, stated here because
it is a decision rather than a detail - it puts the guest serial
console one place further from the budget, and that capture is the
only one describing a worker which never reached apid.

Both reads take the block's own read-budget knob rather than a literal
equal to it, so lowering that knob lowers these too. Each keeps its own
exit status rather than routing through the shared helper, which always
returns zero by design: this collector has to tell "the kubelet was not
read" from "the kubelet answered and carried no worker", and only the
status separates them. The stream is captured whole and filtered after,
since a pipeline reports its last command's status and would otherwise
hand that decision to grep.

Every way a read can come up short is named rather than left as zero
bytes: a kubelet that was not read, one that answered with nothing for
the namespace, a stream cut short, and exit 124 - which cannot
distinguish this collector timing out from the read exiting 124 on its
own - each get their own line, and each says whether an error log
exists to explain it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-12 18:43:11 +03:00
Aleksei Sviridkin e9411d6b8c fix(apps): make the post-delete cleanup hooks report failure 
The qdrant and harbor post-delete hooks guarded every delete with
`|| echo "WARNING: ..." >&2`, set no `-e`, and ended on an
unconditional completion message, so the container exited 0 on
every path. Helm read that as a successful hook and the
hook-succeeded delete policy removed the Job together with the pod
that carried the warning, leaving an uninstall that reclaimed
nothing indistinguishable from one that had nothing to reclaim.

Count the failures instead and set the exit code at the end, so a
failed delete does not skip the deletes that follow it. An object
that is already gone stays a success: every delete keeps
--ignore-not-found, which exits 0 in that case. This is the shape
packages/apps/mariadb already uses for the same problem.

Both hooks also consumed the output of `kubectl get` directly --
harbor by iterating it, qdrant by piping it into grep, whose status
the pipeline then reports. A listing that failed therefore arrived
as an empty list and the sweep was skipped rather than reported.
Each listing is now captured before it is used, so a listing that
failed and one that returned nothing are different events.

A hook that exits non-zero keeps its Job and pods, which is the
point: they hold the log. Give the Job one retry rather than the
Kubernetes default of six or none at all. Every delete is
idempotent, so a second attempt re-issues an API call rather than
re-running any logic, and one short backoff is what separates a
transient apiserver error from a real one -- these failures become
visible for the first time here, and a signal that fires on noise
stops being read, including on the day it is right. Six is the far
end of the same mistake: ten minutes of backoff across seven pods
outlasts the hook timeout, and Helm reports that timeout instead of
the hook's own error.

The qdrant Chainsaw suite explained its post-delete pod collector
by quoting the guard string removed here and concluding the pod is
always collected away. That is now backwards: a failed cleanup is
exactly when the pod survives and the collector is the evidence.
Rewrite the note, and the sentence in the harbor suite pointing at
it.

hack/app-cleanup-hooks_test.bats renders both scripts and runs them
against a fake kubectl. There are eight failure paths across the two
hooks -- six in harbor, two in qdrant -- and each has a case that
goes red when that one path alone is reverted to a swallowed
warning. Message assertions are anchored on the hook's own `ERROR: `
prefix, because the fake kubectl names the object in its own
diagnostic too and an unanchored match is satisfied by the fixture.
The helm-unittest suites pin the retry limit and the hook delete
policy: adding hook-failed to that annotation hands the Job back to
Helm to delete on failure and destroys the evidence this change
exists to keep, and until these asserts that reversal left both
suites green.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-12 18:43:02 +03:00
myasnikovdaniil 516be4abfe fix(cozystack-basics): normalize tenant hostname VAP comparisons with lowerAscii (#3208) 
## What this PR does

The Ingress, Gateway, and Route hostname `ValidatingAdmissionPolicy`
objects compare tenant hostnames against the tenant apex with
case-sensitive CEL. The resource-side host is always lowercase — the
apiserver's hostname validation permits nothing else — but both apex
operands can carry uppercase: the `namespace.cozystack.io/host` label
value, which Kubernetes permits to carry uppercase and which the tenant
chart writes verbatim from `tenant.spec.host`, and the templated
`_cluster.root-host`, a free-form platform value that nothing validates
for case.

A mixed-case apex label wrongly denies a tenant its own legitimate
lowercase in-apex hostname. An uppercase `_cluster.root-host` is the
dangerous direction: the Ingress outside-root exclusion compares a
lowercase host against the uppercase literal, never matches, so the
exclusion stops excluding and the policy admits any non-wildcard host —
every non-wildcard platform-apex name a tenant asks for, including a
hostname belonging to another tenant and the root apex itself. Only the
anti-wildcard gate still holds.

This normalizes both operands of every hostname comparison with
`.lowerAscii()` (host, apex label, and the templated root-host literal)
across all three policies, so the match follows DNS case semantics.
There is no behavior change in the default all-lowercase configuration.

The Gateway policy's fail-open on an absent or empty host label is
deliberately left as it was: that is the platform's one documented
fail-open exception in `docs/security/threat-model.md`, so changing the
posture is a security-model decision rather than a side effect of
normalizing case. Its current shape is pinned by a test.

Test coverage:

- helm-unittest pins each comparison as a whole operand pair — equality
and `endsWith`, across the rule-host and tls-host branches — so dropping
the normalization from any single operand is caught. A pattern matching
a lone `.lowerAscii()` would not: the apex appears four times in the
Ingress expression, so stripping one operand still leaves three matches.
- A rendering case with `_cluster.root-host: Example.ORG` pins the
outside-root escape shut; it is red pre-fix.
- e2e probes set a mixed-case apex on an isolated throwaway namespace
and assert both halves of the match: a lowercase in-apex HTTPRoute and
Ingress are still admitted (red pre-fix, green post-fix), while a
foreign apex and a hostname ending with the apex without a dot boundary
are still denied. Vacuous-pass guards re-read the persisted label and
assert the probe apex really is mixed-case.

Normalizing the apex at the write site — the tenant namespace template
and `_cluster.root-host` — is tracked separately in #3328. It would not
remove the need for this change: the host label is settable out-of-band
by a trusted caller, so these policies must not assume the apex arrives
normalized.

Closes #3205.

### Screenshots

N/A (no UI changes).

### Release note

```release-note
fix(cozystack-basics): tenant hostname admission policies now compare against the tenant apex and the platform root apex case-insensitively. This closes a bypass where an uppercase `_cluster.root-host` made the Ingress policy's outside-root exclusion stop matching, so the policy admitted any non-wildcard host — including a hostname under the platform apex belonging to another tenant, and the root apex itself.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Hostname validation now consistently matches case-insensitively across
Gateway, Ingress, HTTPRoute, and TLSRoute.
* Mixed-case tenant and apex hostnames are accepted when they meet the
intended domain boundaries.
* Foreign domains and prefix-adjacent hostnames remain correctly
rejected; validation behaves as documented when the tenant host is
empty.

* **Tests**
* Expanded unit and e2e coverage for case-insensitive hostname rendering
and boundary rejections across the supported resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-12 15:35:41 +05:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver 
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-08-12 12:00:33 +04:00
IvanHunters b4a2f6f791 Merge remote-tracking branch 'origin/main' into pr-3315-lgtm 
# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go
#	hack/e2e-chainsaw/_lib/run-kubernetes.sh
#	packages/apps/kubernetes-nodes/tests/render-parity.sh
#	packages/apps/kubernetes/README.md
#	packages/apps/kubernetes/templates/cluster.yaml
#	packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml
#	packages/apps/kubernetes/tests/talos_templates_test.yaml
#	packages/apps/kubernetes/values.schema.json
#	packages/apps/kubernetes/values.yaml
#	packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
 2026-08-10 22:11:21 +03:00
Andrey Kolkov d0990299ed feat(clickhouse): let the backup sidecar trust a private S3 CA, unblocking the backup e2e round-trip (#3385) 
## What this PR does

Gives the ClickHouse `clickhouse-backup` sidecar a way to trust an S3
endpoint whose certificate is signed by a private CA, and uses that to
make the ClickHouse backup/restore e2e round-trip actually run in CI.
This actualizes #2600, whose feature code and example scripts have since
landed on `main` independently, and whose remaining contribution — the
e2e test — was still in the retired BATS format. #2600 can be closed
once this merges.

### 1. `backup.endpointCA` on the ClickHouse chart

The sidecar had **no TLS/CA surface at all**: bucket coordinates and
credentials arrive as `S3_*` env, and nothing let it verify an endpoint
signed by a private CA. Cozystack's own in-cluster SeaweedFS is exactly
that case — it serves `:8333` behind the self-signed "SeaweedFS CA" — so
a ClickHouse release could only ever back up to a publicly-trusted
endpoint. MariaDB (`storage.s3.tls.caSecretKeyRef`) and Postgres (barman
`endpointCA`) have had a CA field all along; this closes the gap for
ClickHouse.

`backup.endpointCA` takes `{name, key}`. When set, the chart mounts that
Secret into the sidecar and points `SSL_CERT_DIR` at it. Empty — the
default — renders no env, no mount and no volume, so existing releases
are byte-for-byte unchanged.

**Why `SSL_CERT_DIR` and not `AWS_CA_BUNDLE`.** Both are honoured by
`altinity/clickhouse-backup:2.7.4`; I measured them against a
self-signed endpoint and public AWS S3 in turn. Go reads `SSL_CERT_DIR`
*in addition to* the system bundle, so both endpoints verify.
`AWS_CA_BUNDLE` *replaces* the SDK's trust pool: the private endpoint
verifies, and public S3 then fails `x509: certificate signed by unknown
authority` — it would silently break a release that also talks to a
public endpoint. `S3_DISABLE_CERT_VERIFICATION` was rejected outright,
since it drops verification rather than trusting one more CA.

### 2. The e2e suite, and what it was really testing

Two Tests added to the existing `hack/e2e-chainsaw/clickhouse/` suite
(in name order, mirroring the `etcd`/`mariadb`/`postgres` shape — see
the layout rationale below for why they live here rather than in a new
directory):

- **`clickhouse-1-backup-contracts`** (CI, `tenant-test`, ~98s): a
backup-enabled ClickHouse asserted against the *running* cluster — the
`clickhouse-backup` sidecar on the `ch-backup-api` `:7171` port, and the
`<release>-backup-api-auth` Secret whose keys the sidecar's
`API_USERNAME`/`API_PASSWORD` actually resolve `valueFrom`. Each is a
silent break if a chart bump renames it. Needs no object store.
- **`clickhouse-2-backup-roundtrip`** (CI, `tenant-root`): `Bucket` →
`Altinity` strategy + `BackupClass` → source ClickHouse with a sentinel
row → `BackupJob` → in-place `RestoreJob` → to-copy `RestoreJob` into a
second instance, driving `examples/backups/clickhouse/run-all.sh` so
test and docs cannot drift.

The round-trip was originally gated behind `CLICKHOUSE_E2E_S3_ROUNDTRIP`
and **skipped in CI** — its `PASS (1.01s)` meant the gate fired, not
that a backup happened. `backup.endpointCA` removes the reason for the
gate, so the suite now adopts the same three preconditions
mariadb/postgres rely on: run in `tenant-root` (an isolated tenant's
Cilium egress allowlist blocks its Pods from reaching `tenant-root`'s
SeaweedFS), target the in-cluster endpoint instead of the `Bucket`'s
external ingress URL (an unroutable placeholder in CI), and trust its
certificate via the copied CA. It also pre-cleans with `cleanup.sh`,
because `tenant-root` is shared and a leftover `Succeeded` `BackupJob`
would satisfy the harness's wait without a backup ever running.

Both backup Tests live in the existing `hack/e2e-chainsaw/clickhouse/`
suite rather than a new suite directory, so no selector mapping is
added: `hack/select-e2e.sh` already derives `clickhouse` both from
`packages/apps/clickhouse/**` (the `*-application` rule) and from
`examples/backups/clickhouse/**` (the harness path), and both paths
therefore select the suite that actually runs the harness. That
alignment is the reason for the layout — a separate `clickhouse-backup/`
directory would have made an edit to the harness select a suite that
cannot exercise it. `hack/select-e2e.sh` and `hack/select-install.sh`
are otherwise untouched (the only change is a comment recording that
invariant), and the existing `backup example harness edit selects its
app suite` case in `hack/select-e2e_test.bats` covers the mapping.

### Verification

- `helm unittest` — 17 pass, including 5 new `endpointCA` cases: the
unset default renders nothing, the env/mount/volume wiring, a custom key
still projected as `ca.crt`, composition with `useSystemBucket`, and
`backup.enabled=false` renders nothing.
- The CA-trust behaviour was measured against the real image, not
assumed: without a CA knob the sidecar reports `x509: certificate signed
by unknown authority`; with `SSL_CERT_DIR` the handshake succeeds.
- `hack/select-e2e_test.bats` (16) and `hack/select-install_test.bats`
(17) pass; `select-install.sh --validate` reports graph OK.
- `make generate` is idempotent, so the pre-commit job's `git diff
--exit-code` stays clean.
- The contracts Test passed in CI on earlier runs of this branch. The
round-trip only became CI-live with this work, so it still needs a green
E2E run before merge — the branch has since been rebased onto current
`main` to get one.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff, file by file. The load-bearing
change is the new `backup.endpointCA` field in
`packages/apps/clickhouse/values.yaml` / `values.schema.json`;
everything else is `hack/e2e-chainsaw/`, `examples/backups/`, one docs
paragraph, and regenerated artifacts, none of which appear in the map.

- [ ] No downstream repository is affected by this change
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/18 —
"add, remove or rename a field in an app's `values.schema.json`" reaches
the provider's hand-maintained schema, model and expand/flatten pair, so
the new nested object is not surfaced until it is added there. Opened as
an issue rather than a PR because the natural moment is alongside that
repo's next `api/apps/v1alpha1` pin bump, which is a maintainer call.

Not ticked, with reasons rather than silence: **cozystack/website** —
the trigger is *adding, renaming or removing a package*, and this adds a
field to an existing one. `clickhouse` is already in the website
`Makefile`'s app list and its reference page is generated from this
package's `README.md`, so the regenerated `backup.endpointCA` rows are
carried by the existing release-tag automation with no manual PR.
**cozystack/external-apps-example** — vendors `hack/package.mk` and
`update-crd.sh`, neither of which this PR touches.
**cozystack/ansible-cozystack**, **ccp**, **talm**, **cozyhr**,
**cozy-proxy**, **cozystack-telemetry-server**, **examples** — no
installer values, `hack/` layout, namespace, variant, node-prerequisite,
annotation, label or metric in their trigger rows is touched here.

### Release note

```release-note
feat(clickhouse): add `backup.endpointCA` so the clickhouse-backup sidecar can verify an S3 endpoint whose certificate is signed by a private CA, such as Cozystack's in-cluster SeaweedFS. Leave it unset for publicly-trusted endpoints; existing releases are unaffected.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional private S3 endpoint CA support for ClickHouse
backup/restore sidecars via `backup.endpointCA`, including configurable
Secret name and CA bundle key (default `ca.crt`).
* Sidecars mount the CA and extend trust while leaving public endpoints
unchanged.

* **Documentation**
* Updated ClickHouse chart docs, operational backup guidance, and
backup/restore examples to cover CA setup, overrides, and cleanup.

* **Tests**
* Added helm-unittest and end-to-end coverage validating CA
rendering/mounting behavior and ClickHouse backup/restore workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-10 21:17:47 +04:00
Aleksei Sviridkin ba15b3817a fix(cozystack-basics): normalize tenant hostname VAP comparisons with lowerAscii 
The Ingress, Gateway, and Route hostname ValidatingAdmissionPolicies
compare tenant hostnames against the tenant apex with case-sensitive CEL.
The resource-side host is already lowercase — the apiserver's hostname
validation permits nothing else — but the apex operand
is the namespace.cozystack.io/host label value, which Kubernetes permits to
carry uppercase, and the Ingress policy additionally compares against the
templated root host, a free-form platform value that nothing validates for
case.

A mixed-case apex label wrongly denies a tenant its own legitimate
lowercase in-apex hostname. An uppercase root host is the dangerous
direction: the Ingress outside-root exclusion compares a lowercase host
against the uppercase literal, never matches, so the exclusion stops
excluding and the policy degenerates into admitting any non-wildcard host
— every non-wildcard platform-apex name a tenant asks for, including the
root apex itself. Only the anti-wildcard gate still holds.

Wrap both operands of every comparison in .lowerAscii() so the match
follows DNS case semantics — no behavior change for the all-lowercase
default. Each policy reads a label it does not own: the namespace host
label is settable out-of-band by a trusted caller, so none of them may
assume the apex arrives normalized.

The Gateway policy admits any listener hostname when the host label is
absent or empty, unlike its two siblings. That is the platform's one
documented fail-open exception, recorded in the threat model together with
its compensating control, so changing the posture is a security-model
decision rather than a side effect of normalizing case. This leaves the
behavior untouched and pins its current shape in the tests.

Covered by helm-unittest assertions pinning each comparison as a whole
operand pair, so dropping the normalization from any single operand is
caught; a rendering case that pins the uppercase-root-host escape shut; and
e2e probes that set a mixed-case apex on an isolated namespace and assert
both halves of the match — a lowercase in-apex HTTPRoute and Ingress are
still admitted, while a foreign apex and a hostname that ends with the apex
without a dot boundary are still denied.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-10 01:15:55 +03:00
Aleksei Sviridkin 9d4bcd8d66 fix(e2e): wire the ghcr.io mirror into the tenant install and node-join flow 
Deploy the pull-through mirror from its own install step rather than
folding it into the Talos image-cache step, whose early return on a
missing values key would otherwise skip it silently. Build the tenant
CR spec.talos block from both the OS image cache and the registry
mirror, each included only when its in-sandbox backend is up. On a
node-join failure dump the mirror's state alongside the Talos image
cache, under the same phase gate as its neighbours so it cannot starve
the guest captures of budget. Record the mirror's legs in the step's
budget inventory.

talos_spec_block is covered by a bats suite over all four mirror-up and
mirror-down combinations, splicing the result under spec: and reading
it back with yq so a block indented into the wrong parent fails, and
the install step is pinned to exclude the Cilium policy from its
pre-Cilium apply.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-09 16:17:44 +03:00
Aleksei Sviridkin 1c6ee7c047 fix(e2e): harden the ghcr.io mirror resolver and bound its diagnostics 
resolve_ghcr_mirror_endpoint decides once, for the whole shared
sandbox, whether tenant workers are pointed at the in-sandbox mirror,
so a misclassified transient failure would disable it for the rest of
the run. Cache only stable answers: a genuine API NotFound and a
definite Available=False; never a transient query error, a missing
kubectl, or a failed egress-allow apply. Read readiness from the
Deployment's own Available condition when the rollout watch exits
non-zero, and commit the endpoint only once the egress allow is in
place, surfacing the apply failure reason rather than a generic blip.

The node-join diagnostic counts kubelet-image requests across the whole
access log rather than a fixed tail, and every read it makes is
wall-clock bounded by the caller's COZY_DIAG_READ_TIMEOUT so no single
read can hold the failure path open. The bats suite exercises every
resolver branch and the diagnostic's filtering, with a kubectl stub
that reports errors on stderr as the real binary does and a timeout
mock so the bounded reads stay reachable under test.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-09 16:17:33 +03:00
IvanHunters 58910c0edc feat(kubernetes): route tenant worker image pulls through a registry mirror 
Add a talos.registryMirrors passthrough to the kubernetes and kubernetes-nodes
charts that renders into the worker TalosConfigTemplate machine.registries.mirrors,
so an operator can point a registry such as ghcr.io at an in-cluster pull-through
mirror and a worker boot no longer depends on live public egress. Default {}
keeps the rendered worker reconcile Job byte-identical to before, content-hash
Job name included. Free-form mirror values are shell-escaped for the unquoted
apply heredoc, covered by an execution-level bats test.

Also add the e2e ghcr-mirror (registry:2 pull-through plus a Cilium egress
policy) and wire it into the kubernetes chainsaw suites, so CI exercises the
mirror path and takes the flaky public ghcr.io kubelet-image pull off live
egress.

Refs #3513 (addresses the egress variant of the tenant-worker node-join
failures; the dominant variant is CI-runner CPU starvation, tracked in the
issue).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-09 14:10:16 +03:00
Aleksei Sviridkin 5400f165e3 fix(e2e): keep a failed mirror lookup from pinning the public factory 
resolve_talos_image_factory_url asked whether the in-cluster Talos image
mirror exists with a plain `kubectl get deploy`, where a connection
refused, an Unauthorized and a real absence all arrive as the same
non-zero exit. It then cached that reading for the whole suite, so one
unlucky moment sent every tenant worker to the public Image Factory over
live egress -- the dependency the mirror exists to remove -- while the
run still looked ordinary.

Ask with --ignore-not-found -o name instead: a real absence answers exit
0 with no output, which leaves a non-zero exit meaning only that the
question failed, and re-ask a couple of times before giving up. Only the
empty answer turns the suite toward the public factory. An unanswered
lookup warns, falls back for its own caller alone, and is not written to
the decision file, so the next tenant test asks again rather than
inheriting a guess.

Each attempt is bounded twice, and the two bounds are deliberately not
equal. kubectl defaults --request-timeout to 0, so a connection that
establishes and then stalls has no deadline of its own; the flag alone is
not a wall-clock bound either, because kubectl retries discovery several
times before giving up. Measured against a stub apiserver, the three ways
this call fails behave differently: a refused port answers in under a
second and names the refusal, one that stalls mid-TLS gets kubectl's own
handshake deadline and still produces text, and a blackholed address
produces nothing until some deadline expires. With both bounds at the
same value the expiry that arrives first is always the wall-clock kill,
so --request-timeout can never fire and that last case stays silent. The
wrapper therefore sits ten seconds above it: the request deadline ends
the attempt and kubectl says why, and the wrapper remains the backstop
for a client that overruns its own deadline.

Where timeout is not installed the read runs unwrapped rather than
through it: its exit 127 would otherwise count as an attempt that did not
answer, three times over, and a missing local binary would pin the whole
suite to the public factory while every message blamed the apiserver.

kubectl's stderr is left alone, because it is the only thing separating
refused from unauthorized from stalled. It is still not guaranteed, since
a kill can land before the client has said anything, so the lookup also
names the last attempt's exit status before giving up, in the vocabulary
the diagnostic reads use for 124 and 137.

The diagnostic path asks the same question and keeps its own answer to
it. That one gives up after a single attempt, under a phase budget where
re-asking spends what the snapshot behind it needs; this one runs on the
happy path, where a re-ask costs a suite nothing and buys it the mirror.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-09 13:31:55 +03:00
Aleksei Sviridkin 04494555c9 test(e2e): pin that an unanswered mirror lookup is not a mirror absence 
The Talos image factory decision asks the apiserver once whether the
in-cluster mirror Deployment exists, reads any non-zero exit as an
absence, and caches that answer for the whole suite. A refused, throttled
or timed-out lookup therefore sends every tenant worker in the run to the
public Image Factory over live egress -- the dependency the mirror exists
to remove -- and the run reads as ordinary environment noise rather than
as a decision that was never actually made.

Route that question through one helper and pin what the helper owes its
caller: the three outcomes it must tell apart, the flags that make the
three-way answer possible at all, a deadline on every attempt, the
caching rule -- a real absence is resolved once and cached, an unanswered
lookup falls back for its own caller alone -- and that a missing local
timeout binary leaves the lookup asking the apiserver rather than
answering unknown three times over.

Two of those need saying carefully, because both concern what the caller
is told when the lookup fails. A killed attempt gets its own case, driven
by a wrapper that exits without running kubectl at all, so the lookup has
to account for why it gave up rather than relay someone else's words:
pinning the reason on a kubectl that failed loudly, as the sibling case
does, pins it only where it was never in doubt. And the wall-clock bound
is pinned as strictly greater than the request deadline beside it, which
is the property that lets kubectl reach its own deadline and say what
happened; level with it, the kill always lands first.

The helper is a placeholder in this commit and claims present
unconditionally, so the cases that need the distinction fail.

The stubs are files on PATH rather than shell functions because the
lookup runs under an external timeout binary, which execs a real file and
would never see a shell override. xtrace stays off for the rest of a test
that captures stderr: turned back on, a failing assertion wedges bats
instead of reporting it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-09 13:31:54 +03:00
Aleksei Sviridkin 048088bd29 fix(e2e): bound the tenant failure diagnostics per read and as a phase (#3676) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

The node-join failure diagnostics in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` issued about a dozen
`kubectl` reads with neither an outer wall-clock bound nor
`--request-timeout`, starting with a `describe nodes` through the tenant
kubeconfig. That read targets the component least likely to answer in a
node-join failure. An unbounded read there does not lose only itself: it
holds the Chainsaw op until the op is killed, so everything scheduled
after it is lost rather than truncated, the tenant crust-gather snapshot
above all.

Every read now carries the per-call bound the newer collectors on this
path already use. A read that fails or is cut off says so instead of
passing in silence, and says only what was observed, because a read that
did not finish is silence about the cluster rather than a finding about
it. The CDI importer walk gains a cap, and its listing is read on its
own so a listing that never answered cannot read as a namespace with no
importer in it.

Per-read bounds alone do not end the phase, and the collectors here can
together spend more than what is left of the op by the time they run. So
the phase is bounded as a phase: once its budget is spent, the
collectors that have not started are declined out loud. That is not a
worst case and is not written as one, but it stops the snapshot being
queued behind work begun past the deadline, which is how the snapshot
was being lost. The op ceiling is unchanged, since a larger ceiling
multiplied by a sum that grows with every collector added is the same
problem one collector later.

Because what runs last is what gets declined, the block is reordered so
the cheap reads come first. The CSR list and the signer log are the
discriminator for the failure mode a follow-up fix has to target, and
behind two minute-scale collectors they were exactly what a tight run
would drop.

Adjacent work on the same failure path, all of it in service of the same
requirement:

- the sibling scheduling-gate branch ends in the same exit and depends
on the same snapshot, so its two reads are bounded too;
- `talos_image_cache_diagnose` gets its reads bounded and its gate gets
`--ignore-not-found`, because `kubectl get` exits 1 for a refused
connection or Unauthorized as readily as for NotFound, so a call that
never reached the apiserver was being announced as a mirror that was
never deployed;
- the four numeric knobs are validated rather than only documented,
since a suffix or a leading zero on the budget reached the arithmetic in
the block's first statement and unwound the whole function before its
own headline, leaving a failing run with no diagnostics at all.

Comments carry invariants rather than arithmetic. A comment that states
a sum has to be kept true forever in every copy of it, and any edit to
the code or the order falsifies one of them. The numbers live in the
code as values and in the guard that pins them.

Relates to #3656.

### Known gaps, deliberately not closed here

Some collectors on this path carry no wall-clock bound at all. The
image-cache reachability re-probe makes seven unbounded
management-cluster calls, and the guest-Talos capture walks its VMI rows
uncapped with applies and waits bounded only by `--request-timeout`,
which bounds one HTTP request rather than a client retrying against a
wedged apiserver. The phase budget stops them being started late; it
does not stop one running long once admitted. They are pre-existing and
tracked in #3666. Bounding the re-probe in particular would change when
the same function gives up on the happy path, and so which image factory
tenant workers use, which is not a diagnostic's decision to make.

Named improvements left for later, none of them regressions introduced
here:

- the phase deadline is stamped at block entry rather than against what
is left of the op, so a long bringup still narrows the room the
derivation assumes;
- `bringup` and `largest` are literals in the guard, so a collector
heavier than today's will not move them, and the margin is small by
design;
- the integer-knob validator now exists in four places in this tree, and
a shared `_lib` file sourced ahead of both libraries would collapse two
of them;
- the per-request bound equals the wall-clock bound, so both deadlines
fire together and which of the two notes appears is a race;
- the ordered list in the chainsaw comments omits the ungated wedge
check, which runs first;
- section letters no longer sort after the reorder, and the
comment-to-code ratio in the touched block is high.

### Testing

`hack/run-kubernetes-node-join_test.bats` is new: 22 cases driving the
block with a `kubectl` stub that hangs, refuses and answers in part, and
a `timeout` stub that fails any read not wrapped in the bounding idiom.
It pins that no read escapes a bound, that a cut-off read is named
rather than reported as an empty cluster, that the walk cap reports both
counts, that a failed listing is not reported as no importers, that the
phase declines the rest out loud, that a malformed knob falls back and
says so, that the block returns zero so the caller keeps its `exit 1`,
and the source order that keeps the discriminating reads ahead of the
expensive collectors. Every one was checked by mutation against the
unfixed shape.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(e2e): bound every read in the tenant Kubernetes node-join failure diagnostics and give the collection phase its own wall-clock budget, so a wedged apiserver can no longer hold the Chainsaw op until it is killed and take the tenant crust-gather snapshot with it
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved Kubernetes node-join and scheduling failure diagnostics with
bounded collection times, clearer timeout and failure reporting, and
more reliable preservation of critical snapshots.
- Added safer handling for unavailable diagnostic data, missing tools,
failed commands, and undeployed image caches.
- Added configurable and validated timeout controls for image-cache and
failure diagnostics.

- **Documentation**
- Clarified diagnostic budgets, collection ordering, and behavior when
available time is exhausted.

- **Tests**
- Added comprehensive coverage for diagnostic timeouts, ordering,
validation, failure handling, and bounded reads.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-09 12:33:27 +03:00
Aleksei Sviridkin 86fb92abf5 docs(tests): correct five stale claims in the e2e capture tooling (#3662) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Five things in the e2e tooling said something the code no longer does,
or did not say it at all. All five are prose; no executable line
changes.

`docs/agents/e2e-testing.md` section 8 described two of the three
on-failure capture legs. The missing one is
`hack/e2e-capture-dataplane.sh`, the only leg that reads forwarding
state off the node rather than object state, and the only one whose two
callers bound it differently, 600s from the BATS runner and 300s from
the Chainsaw catch. That split is what decides whether something added
to it fits, so a contributor who reads the document and not the script
sizes a change against the wrong budget. Its notes contract is described
with the carve-out the script declares itself: the two EndpointSlice
reads in the LoadBalancer path report nothing either way, so a `<none>`
backend line there is the one place a reader still cannot tell an
endpointless Service from a read that never answered.

The same section said the runner's captures serve "the BATS tests" in
two places. They serve the e2e ones. The runner keys on an `e2e-`
prefix, on the suite's own filename or on its parent directory, because
all three legs read cluster state and a unit runner has `kubectl` on
PATH with no cluster behind it. The legs are not gated alike either, so
the sentence says which fired where: two key on `kubectl`, and the
snapshot leg keys on `crust-gather` being installed, which on a unit
runner it usually is not.

Two numbers in the capture-gate rationale in `hack/cozytest.sh` no
longer described the code. The snapshot leg's cost was given as 360s,
the deadline passed to `timeout`; the leg is invoked as `timeout -k 30
360`, so the trap waits out 390s, which is what the comment forty lines
below and the op-budget arithmetic in `.chainsaw.yaml` already said. And
the data-plane leg was still described as having no bound on its first
pod list, and billed at the full 600s on that basis, after that read was
given a 28s ceiling.

Two files enumerated the BATS suites and both stopped one short. The
global catch in `hack/e2e-chainsaw/.chainsaw.yaml` said the runner it
names as its analog "only covers the BATS install/openapi tests", and
`hack/e2e-chainsaw/README.md` named the same two files. Both miss
`hack/e2e-prepare-cluster.bats`, which `packages/core/testing/Makefile`
drives through that runner, and which is the suite whose failures leave
the sandbox in the state hardest to reconstruct afterwards.
`docs/agents/e2e-testing.md` already listed all three, so the tree
disagreed with itself depending on which file you opened.

A comment in `hack/e2e-install-cozystack.bats` pointed at
`hack/e2e-apps/run-kubernetes.sh`, which does not exist; that directory
holds two monitoring suites and no runner. The readiness gate it means
is the `rollout status` wait in
`hack/e2e-chainsaw/_lib/talos-image-cache.sh`, along with both fallbacks
to the public factory.

relates to #3626 and #3642


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
NONE
```
 2026-08-09 04:08:35 +03:00
Aleksei Sviridkin 9278702018 docs(e2e): make the op-budget comment name what it guarantees 
The 50m ceiling on the kubernetes-* script ops is justified by a comment
enumerating what the failure path can spend, and the enumeration was
both incomplete and reassuring in the wrong direction. It listed some
collectors, said nothing about the node-join diagnostics or the
image-cache diagnosis, carried a figure for the guest Talos capture that
was half its real cost, and closed by asking the reader to keep the op
above the sum, which no ceiling can do because the sum runs past what is
left of the op on its own.

It now names the phase budget as the thing that keeps the snapshot
reachable and lists the collectors this ceiling covers, in the order the
phase spends it. It states no sums and no per-collector figures. Why the
budget is that number is derived once, beside the budget, and pointed at
from here: a figure repeated across three files is a figure that drifts,
and the drift is what the reader trusts.

It also says what the ceiling does not promise. The budget bounds when a
collector may start rather than when the phase ends; several collectors
on this path carry no wall-clock bound at all; and the bringup's own
waits already exceed this op on ceilings before a single diagnostic
runs. So no worst case is claimed, because none can be stated while that
holds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-09 02:51:06 +03:00
Aleksei Sviridkin 0d7beaf630 fix(e2e): bound the tenant failure diagnostics per read and as a phase 
The node-join diagnostics issued about a dozen kubectl reads with
neither an outer wall-clock bound nor --request-timeout, starting with a
describe nodes through the tenant kubeconfig. That read targets the
component least likely to answer in a node-join failure, and an
unbounded read there does not lose only itself: it holds the Chainsaw op
until the op is killed, so everything after it is lost rather than
truncated, the tenant crust-gather snapshot the failing exit triggers
above all.

Each read now carries the per-call bound the newer collectors on this
path already use, sized lower because this block issues a dozen back to
back where those bound one. A read that fails or is cut off says so
instead of passing in silence, and says only what was observed: a read
that did not finish is silence about the cluster, not a finding about
it. The importer walk gains a cap for the same reason its listing is now
read on its own, since a listing that never answered must not read as a
namespace with no importer in it.

Per-read bounds alone do not end the phase, and the collectors here can
together spend more than what is left of the op by the time they run.
Much of that is not ceiling-bounded at all: on the guest-Talos path the
applies, waits and deletes carry --request-timeout or --timeout, which
bound one HTTP request rather than a client retrying against a wedged
apiserver. So the phase is bounded as a phase. Once its budget is spent
the collectors that have not started are declined out loud. That is not
a worst case and is not written as one, but it does stop the snapshot
being queued behind work begun past the deadline, which is how the
snapshot was being lost. Raising the op ceiling was the alternative and
it is not one: a larger ceiling times a sum that grows with every
collector added is the same problem later.

Because what runs last is what gets declined, the block is reordered so
the cheap reads come first. The CSR list and the signer log are the
discriminator for the failure mode a follow-up fix has to target; behind
a serial-console walk and a guest-Talos capture that can spend the whole
budget between them, they were exactly what a tight run would drop.

The comments carry invariants rather than arithmetic. A comment that
states a sum has to be kept true forever in every copy of it, and any
edit to the code or the order falsifies one of them. The numbers live in
the code as values and in the guard that pins them; the prose keeps why
the budget comes from an inequality, why admission gates the start
rather than the finish, why the cheap reads run first, and which
residuals the guard's literals do not cover.

The sibling scheduling-gate branch ends in the same exit and depends on
the same snapshot, so its two reads are bounded too. So are the reads in
the talos-image-cache diagnosis, whose gate also gains
--ignore-not-found: exit 1 from kubectl get means a refused connection
or Unauthorized as readily as NotFound, and reading it as absent
announced the mirror as never deployed on the strength of a call that
never reached the apiserver. Bounding those dumps needed a note of its
own first, since timeout prints nothing when it fires and a cut-off dump
would otherwise leave a bare section header that reads as a cache with
no Pod, no Service and no EndpointSlice.

The knobs those bounds come from are validated rather than only
documented. Each is pasted somewhere that takes digits and nothing else,
and each failed differently and quietly: a suffixed or leading-zero
budget reached the arithmetic in the block's first statement and unwound
the whole function before its own headline, so a failing run produced no
diagnostics at all; a suffixed read bound became a millisecond request
timeout and made every note blame the cluster for a value; zero disabled
the bound outright; a suffixed importer cap made the comparison exit
non-zero, so the cap stopped existing; and a non-numeric grace makes
timeout exit before running the command at all. A rejected value falls
back and says so, which is what the rest of this tree does with its own
knobs. All four are re-checked where they are used, since a value set
after sourcing never passed the assignment, and the read bound is
checked again before the per-request string is composed from it so the
two halves cannot disagree. Each default is declared once as a constant,
and the re-checks read the bare variable so an unset knob is named
rather than silently defaulted.

When timeout itself is absent the reads run unbounded rather than being
bounded with a binary that is not there: the latter turns every read
into an exit 127 and every note into the cluster refusing, and collects
nothing, where unbounded is what this path did before any of this. The
phase says so once, and says what it costs rather than reassuring.

The reachability re-probe in the image-cache helper stays unbounded and
its comment names all seven of the calls it makes, the fire-and-forget
delete after the probe Pod included: the same function decides the
mirror on the happy path, so changing when it gives up is not a
diagnostic's decision to make. Its bounds are spelled at each call site
rather than precomputed, since a prefix built at source time cannot
follow a knob a test lowers afterwards while the --request-timeout
beside it does.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-09 02:51:06 +03:00
Aleksei Sviridkin c6961541f3 fix(tests): report a data-plane capture cut short by its backstop 
Both callers of the data-plane collector wrap it in a wall-clock backstop
and threw the result away with `|| true`, while the previous-instance
logs leg sitting beside each of them records the exit status and prints
"INCOMPLETE (exit N); kept what landed in ...". When a backstop killed
the collector, nothing said so, and a truncated dataplane/ directory read
exactly like a complete one that found little.

The collector itself now names every read of its own it could not finish,
which left these two call sites as the only remaining way for that leg to
die without a word. The Chainsaw one is the tighter of the pair -- 300s
against 600s, because it shares an op envelope with the snapshot leg --
so it is both likelier to fire and more expensive to leave silent. In
each place the status is read, never propagated: a diagnostic still
cannot change a test's outcome.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-08 17:08:54 +03:00
Aleksei Sviridkin b11066e770 docs(e2e): name every suite that stayed on the BATS runner 
Two files enumerated the BATS suites and both stopped one short.

The global catch in .chainsaw.yaml describes itself as the Chainsaw
analog of the EXIT-trap host snapshot in hack/cozytest.sh, and said
that runner "only covers the BATS install/openapi tests". The
README said cluster bootstrap and the OpenAPI checks remain BATS and
named only two files. Both miss e2e-prepare-cluster.bats, which
packages/core/testing/Makefile drives through the same runner, and
which is the suite whose failures leave the sandbox in the state
hardest to reconstruct afterwards.

docs/agents/e2e-testing.md already listed all three, so the count was
answerable from the tree; it just disagreed with itself depending on
which file you opened.

The .chainsaw.yaml paragraph is reflowed because the correction is
longer than the line it replaces; no other wording changed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-08 11:49:15 +03:00