<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Fixes#3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.
One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (f4aef7305 re-vendors
`linstor-scheduler` to upstream chart 0.3.1 / extender v0.3.6, which
keeps the webhook and stops stripping unknown pod fields), so after the
rebase this PR no longer touches `linstor-scheduler`; just do not hold
that package back on an older version when rolling out the plugin.
Supersedes #3306 (the keycloak `system`-image workaround): keycloak-db
now backs up via the plugin on the `standard` image, so that stopgap is
no longer needed.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: docs/operations/backup-classes.md changed; a website mirror
follow-up is warranted (not yet opened).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(backups): migrate CNPG Postgres backups to the CloudNativePG Barman Cloud plugin
Postgres backups (apps/postgres, system/keycloak, and the cozy-default-cnpg BackupClass strategy) now use the CloudNativePG Barman Cloud plugin (spec.plugins + a barmancloud.cnpg.io/ObjectStore, method: plugin) instead of the deprecated native spec.backup.barmanObjectStore (removed in CNPG 1.29; the barman-cloud-* binaries it needs are absent from the standard image variant keycloak-db pins, and the bare system-flavor tags apps/postgres pins are deprecated upstream). The barman-cloud plugin operator is installed alongside the CNPG operator.
One platform-level change rides along and requires operator attention on upgrade:
- The CloudNativePG operator is bumped to 1.28.1 (fleet-wide minor upgrade); this also fixes an operator-newer-than-its-CRDs skew that made the operator report "instance manager was restarted during backup" and fail every backup. The vendored chart ships the CNPG CRDs as ordinary release manifests (crds.create), so helm upgrade updates them with the operator; the skew came from the image.tag pin outrunning the chart's CRDs, which this PR removes.
- The plugin's restartable sidecar requires the linstor-scheduler admission webhook that no longer strips initContainers[].restartPolicy (upstream chart 0.3.1 / extender v0.3.6, shipped separately on main); do not hold the linstor-scheduler package back on an older version when rolling out the plugin.
```
Close#3300, #3246
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Switched PostgreSQL CNPG backups/restores to the plugin-based Barman
Cloud workflow, using dedicated ObjectStore resources for backup and
recovery.
* Added the Barman Cloud CNPG-I plugin chart (deployment, TLS, RBAC,
leader election).
* **Bug Fixes**
* Improved backup/restore attachment messaging when the CNPG Cluster
isn’t ready yet.
* Preserves server-name behavior to avoid WAL-archive prefix issues;
clarified endpoint CA handling.
* **Changes**
* Helm integration no longer uses the legacy
`spec.backup.barmanObjectStore` path; updated gating, credential
projection, and ObjectStore-related RBAC/cleanup.
* Removed the linstor-scheduler admission webhook; updated
CloudNativePG/operator versions and CRDs.
* **Tests**
* Updated controller and Helm-unittest checks for the plugin flow;
added/extended end-to-end backup/restore coverage.
* **Documentation**
* Expanded upgrade/migration and backup documentation; updated backup
example materials and comments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The backup roundtrip e2e copied the seaweedfs S3 CA from a Secret named
seaweedfs-system-ca-cert in tenant-root, a name captured off a dev
cluster. On current main the seaweedfs chart pins
fullnameOverride: seaweedfs, so the cert-manager CA Certificate's secret
is seaweedfs-ca-cert; the old name is absent and the first CI run of the
e2e failed at step 05a with 'secrets "seaweedfs-system-ca-cert" not
found'.
Default S3_CA_SECRET to seaweedfs-ca-cert and, when that Secret is
absent, auto-discover the seaweedfs CA Certificate (the seaweedfs-labelled
one with spec.isCA=true) and read its secretName, so a future
fullnameOverride change does not silently break the copy again. Fail with
a clear message when neither resolves. README and chainsaw comment
updated to match.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
The etcd suite's metrics contract failed in CI with `metrics endpoint
did not return 200 over http on the 'metrics' port; got: If you don't
see a command prompt, try pressing enter.`
The probe runs a one-shot curl Pod and reads a `code=<status>` marker
back off its output via `kubectl run --rm --restart=Never --attach`.
`--attach` only establishes its connection once the Pod reaches Running,
so a container that exits in milliseconds — which is what curl against a
Pod IP does — writes its stdout into the void, and `--rm` then destroys
the Pod carrying the log that still held it. The step reports the
metrics endpoint as broken on a cluster that was serving it correctly
throughout.
The evidence that this was output loss rather than an endpoint failure
is that the captured output contained no `code=` at all. A genuine
connection failure would still have printed `code=000` from `-w`, plus
curl's own error text from `-fsS`, and both were being captured.
Meanwhile the probe Pod's events show a clean run, all three etcd
members were up without restarts, and every preceding assert in the
suite passed.
The fix drops `--rm`, re-reads the Pod log when the attach stream
carried no marker, and deletes the Pod explicitly afterwards. This is
the same treatment `hack/e2e-chainsaw/_lib/talos-image-cache.sh` already
applies to the identical race — that probe documents it inline; the etcd
one was simply never given it. A `--pod-running-timeout` is set for the
same reason it is there: the probe image is used nowhere else in the
tree, so it pays a cold pull on every run and kubectl's one-minute
default was a second way to end up with no marker.
The recovery cannot turn a real failure green. The marker encodes the
outcome, so a `code=503` recovered from the log still fails the step; an
attach stream that already carried a marker is never second-guessed; and
the pre-run delete blocks, so a stale Pod's log cannot be read in place
of this run's. Verified by running the probe's exact curl invocation
against a 503 and against a dead port: the first yields `code=503`, the
second `code=000` plus curl's error, and both fail. The probe still runs
exactly once — this is output recovery, not a retry.
Helper extracted to `hack/e2e-chainsaw/_lib/etcd-probe.sh` with unit
coverage in `hack/etcd-probe_test.bats`.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved etcd metrics probe reliability by recovering probe results
when attached output is incomplete.
* Preserved meaningful probe and Kubernetes error details, including
non-success status codes.
* Improved probe pod timing and cleanup to reduce intermittent
end-to-end test failures.
* **Tests**
* Added coverage for successful recovery, failed probes, missing output,
and log retrieval errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The `kubernetes-previous` suite failed with `timed out waiting for the
condition on helmreleases/kubernetes-test-previous-version-csi`, and the
reported cause looked like a dependency problem — `csi` waiting on
`vsnap-crd`, `coredns` waiting on `cilium`. It was neither. The cluster
snapshot taken at failure shows the tenant `kube-apiserver` containers
terminated with exit code 137 within about two minutes of each other:
one killed for failing its liveness probe as reclaim stalled it, the
other OOMKilled outright, both against `limits.memory: 512Mi`.
The e2e fixture pinned the tenant control plane to the legacy `small`
preset — 1 CPU and 512Mi — while the chart ships `c1.medium`, 2 CPU and
2Gi, as the default for this field. A two-node tenant cluster running
the full addon set opens enough watches to cross 512Mi once the workers
join, so the apiserver restarts, and every in-flight HelmRelease that is
polling a DaemonSet rollout burns its entire ten-minute budget while the
control plane is unavailable. That is why two unrelated releases, `csi`
and later `ingress-nginx`, failed with the same signature while the
Deployment-based releases that had installed earlier succeeded. The
`DependencyNotReady` lines are aggregated first-occurrence events —
`vsnap-crd` had in fact installed successfully nine minutes before
cilium did.
This is a defect in the fixture rather than in the product: users on
defaults get four times the memory. The default was raised deliberately
in v0.40.4, "to ensure more reliable operation under higher workloads
and prevent resource constraints"; the fixture was left behind on an
unexplained downward override. Setting it to the chart default makes the
test exercise a control plane sized the way a real one is.
No timeout is widened and no retry is added, since either would hide the
cause rather than remove it. The sibling OIDC fixtures keep the smaller
preset, correctly — they run a single replica with no workers, where
this failure mode cannot occur. Suites are executed serially, so the
additional memory applies to one tenant cluster at a time.
What this does not prove is that 2Gi is empirically sufficient; that is
the value the chart itself ships, so if it still proves too small, the
finding belongs to the product default rather than to this fixture.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Updated the Kubernetes test environment’s control plane resource
allocation to improve reliability and reduce out-of-memory errors and
timeouts.
* **Documentation**
* Added guidance explaining the resource selection and previous failure
conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The tenant Kubernetes e2e pinned controlPlane.apiServer.resourcesPreset to
the legacy "small" preset, capping the apiserver at 1 CPU / 512Mi -- a
quarter of the memory the chart itself ships as the default (c1.medium,
2 CPU / 2Gi).
512Mi does not hold a two-node tenant cluster running the full addon set.
Once the workers join, cilium, coredns, metrics-server, csi, ingress-nginx
and VPA all establish watches at roughly the same moment, and the apiserver
crosses the limit: one replica is killed for failing its liveness probe as
reclaim stalls it, the other is OOMKilled outright. Both replicas restart
within about two minutes of each other, so any tenant HelmRelease that is
mid-install and waiting on a DaemonSet rollout observes 'InProgress' for
its entire timeout and fails, then re-enters a fresh install on remediation.
Which release loses is a matter of timing rather than of the release, which
is why the failure moved between the csi, ingress-nginx and cilium releases
and looked like an unrelated dependency problem each time.
Use the chart default so the test exercises a control plane sized the way a
real one is, instead of one held at a quarter of the shipped default.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The metrics contract runs a one-shot curl Pod and reads a `code=<status>`
marker back through `kubectl run --attach`. Attach connects only after the
Pod reaches Running, so a container that exits before the connection is up
writes its stdout into the void: the step then sees only kubectl's own
"If you don't see a command prompt, try pressing enter." and reports the
metrics endpoint as broken. curl against a Pod IP finishes in milliseconds,
so losing that race is routine, and `--rm` compounded it by deleting the Pod
the instant it exited, taking the log that still held the real output.
The result was a false negative on a healthy cluster, non-deterministic in
the way that reads as a flake because whether attach wins depends on Pod
startup timing rather than on anything under test.
Drop `--rm`, keep the Pod until its result is in hand, and re-read the log
when the attach stream carried no marker. The recovery cannot mask a real
failure: the marker encodes the outcome, so a recovered `code=503` stays a
failure, and a stream that already carried a marker is never second-guessed.
The pre-run delete stays blocking so the re-read cannot find a previous
incarnation's marker, and the post-run delete is guarded so a transient API
error cannot abort the step with the result in hand but unreported.
Pin `--pod-running-timeout` at 90s rather than leaning on kubectl's 1m
default. curlimages/curl is not otherwise used on these nodes, so the probe
pays a cold pull, and a pull that outruns the budget produces the same
markerless output this change exists to explain.
The resolver is split into a sourced helper so the recovery is unit-tested
without a cluster; the kubectl orchestration around it stays covered by the
e2e run. The image-cache probe carries the same recovery inline for the same
race.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The suite allowed 3 minutes for a MariaDB endpoint address to appear,
which was set when a replica had 40 seconds to finish its first boot.
The startup probe now grants 310 seconds, so a healthy install can
legitimately take longer than the gate allows and would be failed while
the probe is still willing to wait.
The endpoint assert is the gate that matters here. The chart hands the
operator a MariaDB custom resource and ships no built-in workload, so
helm has nothing to wait on and the HelmRelease reports Ready as soon as
the manifests are applied, long before a server exists. An endpoint
address, by contrast, appears only once a replica has passed its startup
and then its readiness probe, so that assert is what actually covers
bootstrap.
Raise it to 10m in both the replicated and the single-replica test. One
ready address satisfies the assert, so it has to clear a single startup
budget rather than one per replica, plus the operator reconcile, PVC
bind and image pull that precede it. The HelmRelease ceilings stay at
6m, since nothing about them gates on the server coming up.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Commit 3dbc7d366 removed ExposureClass/ServiceExposure (api/network, the
controller, and the CRDs shipped in cozystack-controller/definitions) in
favor of the native LoadBalancer path, but the chainsaw port that landed
the next day (dd2f89d20) brought the serviceexposure suite along. The
suite creates network.cozystack.io/v1alpha1 fixtures against an API that
no longer exists, so every full-suite run (any PR touching core paths,
and every release cut) fails its two tests with "no matches for
network.cozystack.io/v1alpha1".
Remove the dead suite and its wiring: the suite_to_source case and its
core-platform bats test in select-install (the "-" mechanism stays for
the next core-platform-owned suite), and the README suite list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Ungate postgres-2-backup-roundtrip: instead of an opt-in env flag it now
runs whenever TIA selects the postgres suite (postgres changes, and every
release cut), like the other chainsaw tests. The blocker was network
placement, not capability: the S3 client is the barman-cloud sidecar
inside the Postgres Pod, and the isolated e2e tenant's Cilium egress
allowlist blocks it from tenant-root's seaweedfs — so the Test runs in
tenant-root (same-tenant egress) against the in-cluster S3 endpoint,
trusting its self-signed cert via the copied seaweedfs-system-ca-cert.
This is exactly the recipe validated end to end on a real cluster.
Also map examples/backups/<app>/ edits to that app's suite in
hack/select-e2e.sh (the round-trip executes those scripts as its
harness; previously such edits were silently ignored), with bats
coverage for both the mapped and the no-suite case.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Port examples/backups/postgres into an automated e2e, following the etcd
backup precedent: the numbered example manifests become a runnable harness
(00-helpers.sh, run-all.sh, cleanup.sh) that a human reads and the test
drives, so the documented flow and the test can never drift.
run-all.sh provisions a Bucket, deploys the source Postgres (writing a
sentinel row), derives the two Secrets the barman-cloud ObjectStore
references from the Bucket's BucketInfo and seaweedfs' CA, submits an
ad-hoc BackupJob and waits for Succeeded, then restores to a copy and
asserts the sentinel round-tripped through S3 — the in-cluster witness
that a plugin backup is both taken and restorable end to end.
A second Test document (postgres-2-backup-roundtrip) in the existing
hack/e2e-chainsaw/postgres suite drives run-all.sh, gated out of CI behind
POSTGRES_E2E_S3_ROUNDTRIP=1 (kind has no reachable, CA-trusted S3 endpoint,
and a tenant Pod cannot cross the Cilium egress policy to tenant-root's
seaweedfs). Placing it in the postgres suite (rather than a new dir) keeps
it triggered by packages/apps/postgres changes under Test Impact Analysis.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300):
- docs/operations/backup-classes.md: add an operator-facing upgrade note for the
two platform-wide changes this migration carries — the CloudNativePG operator
bump to 1.28.1 (with the CRD-skew fix and the Helm-CRD-upgrade caveat) and the
LINSTOR scheduler admission-webhook removal, which makes data-locality
scheduling advisory cluster-wide (spec.schedulerName no longer auto-assigned;
opt in via explicit schedulerName). These affect more than Postgres backups and
were previously only noted in a chart values.yaml comment.
- hack/e2e-chainsaw/postgres/postgres.yaml: drop stale restic-era backup fields
(s3Region, s3Bucket, cleanupStrategy, resticPassword) that no longer exist in
the Postgres backup schema; use current fields (destinationPath, endpointURL,
retentionPolicy, 6-field schedule).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Non-blocking review notes from #2826:
- Add `# shellcheck shell=bash` to the four sourced `_lib/*.sh` helpers
(etcd-cleanup, remediation-guard, run-kubernetes, talos-image-cache) so
shellcheck stops flagging SC2148 (no shell directive) — they are only ever
`. sourced`, so there is no runtime interpreter to infer from a shebang.
- Add a `# TODO(chainsaw>0.2.15)` marker at the postgres Service asserts
explaining the `(ports[0].port)`/`(ports[*].port)` projection is a v0.2.15
number-literal-filter workaround (and that `ports[0]` checks only the first
port), so it gets reverted on a chainsaw bump; cross-references the same
workaround in kafka/mariadb/harbor and the e2e-testing.md gotchas section.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).
Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.
Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
tenant-test -> mirror pod in kube-system), applied at point-of-use once
Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
must get 206 - the exact egress path a real importer faces; tenants use the
mirror only when that end-to-end check passes, else fall back to the public
factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
strict 206 gate, and the --overrides JSON builder.
Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main's #3233 landed mid-reconciliation and touched the two OIDC bats
again: nodeGroups (and storageClass) dropped from the render-side test
CRs — the schema default {} now renders one zero-replica
MachineDeployment via CAPI defaulting, so the tests provision zero
worker DataVolumes (#3231). Resolve the modify/delete as deleted and
mirror the fixture change (and its rationale) into the ported
kubernetes-oidc-system / kubernetes-oidc-customconfig Chainsaw suites.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
run under set -eu, so one transient delete failure skipped the
remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
pf_pid (script can exit before the port-forward starts)
Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added securitygroup.bats (#2922 membership model) and
serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the
Chainsaw migration branch was cut.
- securitygroup: two Tests — the aggregated-API projection round-trip
(backing CNP marker label, empty-value membership endpointSelector,
lineage fromApp + membership fromSG peers, 1:1 rule translation, the
view hiding storage internals, unmarked-CNP invisibility via an error
op, delete propagation) and the controller membership-label
stamp/clear lifecycle. Only the attachments annotation stays a jq
script (it is a JSON string). Chainsaw auto-cleanup covers every
fixture even on mid-test failure — the bats had no teardown hook and
leaked its fixtures.
- serviceexposure: the externalIPs backend resolve/Ready/assignedIPs
path with the no-MetalLB-pool negative as an error op (ordered after
resolvedBackend so it cannot false-pass), and the finalizer-removal
test collapsed onto the delete op itself, which waits for full
disappearance.
- selection: kubernetes-application now also maps to the two OIDC
render-side suites (they exercise the same app chart — a gap main's
bats selection had too); the new dirs enter all_apps discovery
automatically. Selector unit tests extended and green (13/13 under
cozytest).
- README: suites list updated.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added kubernetes-oidc-system.bats and
kubernetes-oidc-customconfig.bats after the Chainsaw migration branch
was cut; without a port the 260 lines of coverage would vanish with
the hack/e2e-apps/ deletion.
Both are light cluster e2e (apply CR -> assert rendered objects; no
Kamaji bringup, minReplicas: 0), exactly Chainsaw's declarative shape:
- System mode: KamajiControlPlane carries --authentication-config +
the chart-owned volume/mount; the AuthenticationConfiguration Secret
carries the cozy-realm issuer (regex, script step) + per-cluster
audience; per-cluster KeycloakClient/KeycloakClientScope land in the
cozy realm
- CustomConfig: the flag is still wired; the Secret carries the
tenant-supplied issuer and NOT the cozy realm (fail-loud negation);
no Keycloak objects and no pre-rendered OIDC kubeconfig Secret,
expressed as error ops ordered after the positive asserts so an
unreconciled HR cannot false-pass them
The port also sheds two latent bats bugs: cozytest defines no skip()
(the Keycloak guard would have died with exit 127 had it ever fired)
and the api-resources group probe exits 0 even when the group is
absent — the chart capability-gates Keycloak rendering and CI always
ships the operator, so the asserts are unconditional here.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- qdrant: add the post-delete PVC-reclaim guard (#3059) — delete the CR
and poll the ordinal-anchored data PVC to absence via the error op
(retries transient API errors internally), with a PVC/jobs/pods dump
on failure; Chainsaw auto-cleanup alone would never notice a leaked
StatefulSet-templated PVC
- kuberture: make the split-horizon negative log assertions fail loud —
under set -e a pipeline beginning with ! suppresses errexit, so both
leak checks silently passed (the same vacuous-negation bug main fixed
in 12fe3256f)
- foundationdb: retry the one-shot generations.reconciled read for up
to 2m — the operator stamps it after the post-bounce uptime gate, so
it can trail the fdbcli-derived health fields the previous assert
converged on (a79f8009f)
- kubernetes-latest: add the version-independent SC-fallback-default
Test (#2872 B1) calling verify_storageclass_fallback_default from the
shared lib; a separate Test doc so it runs regardless of the heavy
bringup Test's outcome, safe under the suite-wide parallel: 1 while
it temporarily mutates cluster-scoped StorageClasses
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fold main's post-branch bucket/harbor bats drift into the Chainsaw
suites:
- BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m
default, harbor a stale 10m with the reversed 'can lag on a loaded
runner' rationale): the objectstorage-controller requeues the claim
until the backend Bucket is ready (0f1cc548b), so readiness converges
in tens of seconds and a claim not ready after 2m is frozen — the
tight bound surfaces the propagation regression instead of masking
it (63d14e748)
- the bucket viewer-upload soft-xfail is promoted back to a hard
failure: accessPolicy=readonly enforcement landed with
seaweedfs-cosi-driver v0.3.1 (#1921, eab9b034f), so a successful
readonly upload is now a regression, exactly what the suite's own
TODO called for
- both catch blocks gain the backend view (cluster-scoped Buckets +
cozy-objectstorage-controller pods) where a frozen claim actually
surfaces
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main's mariadb.bats gained "Create single-replica MariaDB" after the
Chainsaw port was cut: replication is guarded on replicas>1
(c7e8f4d97), so replicas=1 must render a CR the operator webhook
admits and provision the bare <name> Service without the
-primary/-secondary split. Mirrors the bats assertions (svc :3306 +
endpoints, StatefulSet replicas=1, metrics svc/endpoints/deployment)
and dumps the rendered k8s.mariadb.com CR in catch so a webhook
rejection is legible.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Main added cozystack-ingress-hostname-policy (d89ad7069) with 12 bats
cases after the Chainsaw port was cut; this appends them as one Test.
- VAP/binding installation asserted declaratively (binding must carry
Deny)
- rejects and accepts grouped by derivation needs: apex-derived rejects
(foreign-apex host, dot-boundary prefix-adjacent, foreign tls host,
wildcard matching the root apex), static catch-all rejects (hostless
rule, empty-string host, defaultBackend), own-apex accepts (in-apex
rule+tls, own-apex wildcard), and static external-domain accepts as a
declarative apply with auto-cleanup
- hostnames derived at runtime: the tenant apex from the test
namespace's namespace.cozystack.io/host label (suite convention) and
the platform ROOT apex from the same label on tenant-root, replacing
the bats' hardcoded test.example.org/example.org; evil${apex}
preserves the dot-boundary semantics of eviltest.example.org, and the
external-domain probes use the RFC 2606 .invalid TLD so they can
never collide with a real cluster apex
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The suite was ported from the pre-#2859 etcd.bats and asserted
etcd.aenix.io/v1alpha1 objects that no longer exist on main — every
Test hard-failed at the first assert on an unknown GVK.
- drop the etcd-empty-backup and etcd-backup-schedule scenarios with
their fixtures: the chart no longer renders EtcdBackupSchedule or
etcd-s3-creds (backups moved to the generic backups API), and
legacy-values tolerance is pinned by helm-unittest, not e2e
- rewrite the remaining scenario into etcd-1-contracts, mirroring
main's contracts test: gate on status.readyMembers=3 (Available=True
flips at seed-member discovery and races replica-count contracts),
assert the operator-owned member Pod shape (strict selector,
container name, named metrics port), WorkloadMonitor
operational/availableReplicas, and script the three contracts that
need imperative probes: the /scale subresource, the plaintext-HTTP
metrics probe against the Pod IP, and the on-demand defrag Job
- add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh,
gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no
publicly-trusted S3 endpoint the snapshot agent could verify)
- upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/
EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs
- strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe
the Etcd CR itself (it can linger Terminating and block the next
singleton apply) and use the err-sentinel pattern so a transient API
error is never misread as "deleted"; add etcd_backup_cleanup wrapping
examples/backups/etcd/cleanup.sh for the round-trip's teardown
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed
test-apps-% target; switch it to the single chainsaw invocation
(empty CHAINSAW_SUITES = full suite) and upload the JUnit report
- pin the chainsaw-report upload-artifact step to the same SHA the
rest of the workflows use (scorecard/zizmor gate)
- repoint hack/run-kubernetes-drain_test.bats at the relocated
hack/e2e-chainsaw/_lib/run-kubernetes.sh
- replicate cozytest.sh's NotReady-pod data-plane capture
(hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so
chainsaw-suite failures keep diagnostic parity with the BATS lanes
- refresh stale bats path comments
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The etcd suite still failed every run after the clean-after-each rework:
etcd-empty-backup (first to run) passed, but etcd-basic and etcd-backup-schedule
wedged at EtcdCluster WaitingForFirstQuorum until the chart's helm --wait timed
out (InstallFailed).
Root cause, from the failed run's crust-gather snapshot: the StatefulSet's
data-etcd-{0,1,2} PVCs are retained on uninstall and the drain never reclaimed
them, so each later scenario rebound the first scenario's volumes (identical PV
UIDs and a single creationTimestamp across all three scenarios) and booted the
previous etcd's data dir and member identity — never forming a fresh first
quorum. The pods were individually Ready (StatefulSet readyReplicas: 3) with all
certs valid, which is why it looked like an operator/network issue rather than
stale storage.
Delete the data PVCs (app.kubernetes.io/managed-by=etcd-operator) in
etcd_cleanup and wait for them to clear alongside the HR and DataStore, so each
scenario starts from empty storage.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The viewer (readonly) S3 credential can still upload: the seaweedfs-cosi-driver
does not yet honor accessPolicy=readonly (cozystack/cozystack#1921, blocked
upstream on kubernetes-sigs/container-object-storage-interface#244). The whole
cozystack chain is correct — Bucket.spec.users.viewer.readonly: true binds the
viewer BucketAccess to the tenant-root-readonly BucketAccessClass whose
parameters.accessPolicy is readonly — but the driver grants the identity write
access anyway (verified live on dev10). The BATS original asserted the same
denial; its retries and toothless gate hid the failure, which this migration
surfaces.
Keep exercising the path and verifying admin read/write + viewer read, but emit
a ::warning:: instead of failing when the viewer can upload, so the suite is not
blocked on an unimplemented upstream feature. TODO(#1921): restore the hard
failure once the driver enforces read-only access.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- mongodb/redis: assert status.readyReplicas instead of status.replicas, so the
StatefulSet check verifies pods are actually Ready (matches openbao/qdrant).
- kuberture: poll for the asynchronously-populated target annotation before
asserting the annotation shape (it was a one-shot test under set -e that could
race on a slow cluster), and guard the probe-Pod phase capture so the
describe/logs diagnostics still run when the Pod never materializes.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd scenarios share the singleton 'etcd' release and relied on the
next scenario's ensure-clean to drain the previous one. Chainsaw runs Tests in
name order (not file order), and v0.2.15 has no test-level finally, so the
last-run scenario leaked its etcd plus the wedged Kamaji DataStore into every
later suite on full/release runs (lexfrei review B3).
Make each scenario self-contained instead: a single run step pre-cleans any
leftover etcd, installs and asserts, then drains in a finally that always runs
(the step is always reached, so teardown fires on success and failure alike).
Extract the shared drain to _lib/etcd-cleanup.sh, removing the ~25-line block
that was copied across all three scenarios.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw port capped the harbor-test-registry BucketClaim bucketReady wait
at the default 5m, half the 600s the BATS original used. seaweedfs/COSI
provisioning can lag several minutes on a loaded runner, so the dependent
harbor-test-system HelmRelease keeps its 10m budget while the BucketClaim it
depends on was capped at 5m — a latent false-flake. Restore 10m
(docs/agents/e2e-testing.md section 6).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The openbao suite's finally deleted PVC data-openbao-test-0 while the
StatefulSet pod still mounted it; the kubernetes.io/pvc-protection finalizer
blocked removal and the delete hit its 5m timeout, failing the suite on every
run (confirmed in CI runs 28084122948 and 28025182013). Delete the OpenBAO CR
first so the HelmRelease/StatefulSet/pod tear down and release the PVC, then
remove the now-unmounted PVC — mirroring the BATS teardown order.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
run_kubernetes_test applies the tenant Kubernetes CR imperatively (kubectl), so
Chainsaw's auto-cleanup — which only tracks resources it apply/create-d — never
reclaims it. On success the script's tail deleted it, but on failure nothing did:
cozy_cleanup() used to be invoked by hack/cozytest.sh's trap, and the Chainsaw
harness has no equivalent caller, so a failed kubernetes-latest/previous run
leaked the tenant cluster (tens of GiB of worker PVCs) into the shared
tenant-test namespace and starved later suites.
Add a `finally` block to both kubernetes suites that runs cozy_cleanup. `finally`
always executes, after the `catch` phase, so the ordering matches the BATS
harness: pass -> cleanup; fail -> crust-gather (tenant trap + host catch) ->
cleanup. cozy_cleanup stays best-effort (|| true) so a slow tenant teardown never
flips a passing test red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.
Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.
The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Convert hack/e2e-apps/backup.bats.disabled to a Chainsaw test under
hack/e2e-chainsaw/backup/. It stays opt-in: the chainsaw-test.yaml.disabled
filename keeps it out of chainsaw's default discovery and out of
hack/select-e2e.sh enumeration, matching the previous disabled state.
The declarative readiness chain (Bucket -> VirtualMachine -> BackupJob) becomes
assert operations; the imperative Velero-backup + S3 verification stays a single
script step, mirroring the bucket suite. README documents the opt-in run command
and the backup-infra preconditions.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The three etcd Chainsaw scenarios share the singleton release name 'etcd' and
the cluster-scoped DataStore/<namespace> the etcd chart provisions for Kamaji.
On teardown Kamaji leaks its kamaji.clastix.io/TenantControlPlane finalizer on
that DataStore — once the backing etcd is gone the reconcile that would drop it
cannot complete — so the DataStore wedges in Terminating and the etcd
HelmRelease uninstall hangs. In CI this errored every etcd case: the first
scenario's stuck teardown blocked the next scenario's install (HelmRelease
never Ready, "DataStore status: Terminating", etcd pods refusing :2381 because
the CA issuer had no key material).
Make teardown explicit instead of relying on Chainsaw auto-cleanup (which was
waiting on the wedged uninstall): set skipDelete on each Test and have every
scenario's ensure-clean drain the previous one — delete the etcd and
force-clear the orphaned finalizer (no TenantControlPlane uses this
per-namespace test DataStore) in a loop until the HelmRelease and DataStore are
both gone. Each scenario then installs from a clean slate.
Verified on a dev cluster: all three scenarios pass back-to-back where they
previously cascaded.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.
- Declarative ports assert on status/conditions and concrete fields; the
repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
collapses into a single polling assert with structured failure diffs and
automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
init/unseal, kuberture split-horizon probes, vminstance, the gateway
admission/impersonation cases, and kubernetes-latest/previous, which wrap
the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
runtime, so they are host-independent.
CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.
Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The (conditions[?type == 'Ready'])[0] indexed-filter idiom fails in
chainsaw v0.2.15 with 'field not found in the input object'. Use the
filter-as-list form instead. Verified: both tests pass end-to-end
against a live cluster.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Port two representative BATS E2E tests (postgres, bucket) to Kyverno
Chainsaw as a pilot for replacing the bash-based suite. Declarative
asserts replace the timeout/until polling boilerplate, catch blocks
add automatic failure diagnostics, and script steps cover the
imperative probes (psql, mc S3 access checks).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
