Commit Graph
6309 Commits
Author SHA1 Message Date
Andrey Kolkov 02bf3e9970 fix(backups): probe live Kafka topics without a pipe and isolate the scratch file
The restore existence check piped the live topic list into `grep -qxF`.
grep exits at the first match, so once the matched topic is far enough from
the end of a list larger than the 64 KiB pipe buffer, the producing printf
takes SIGPIPE, pipefail turns the pipeline into a failure, the `if` reads
false, and a topic that exists is sent to --create, which fails with
TopicExistsException. Read the list from a here-string instead: no pipe, no
SIGPIPE, decided by content rather than list size.

Also make the local metadata scratch file overridable (METADATA_FILE),
like BIN already is, so the bats suite can point each test at its own path
instead of sharing /tmp/kafka-metadata.txt across runs. The Job never sets
it and keeps the default.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00
Andrey Kolkov ca2a6d6429 fix(backups): warn when a Kafka backupTimeout is rejected
kafkaRunDeadline silently substituted the 30m default for a backupTimeout
that time.ParseDuration rejects or that falls below the one-minute floor,
with no Event, condition or log line. An operator who set an unparseable
or sub-minute value, then raised it, saw every run keep dying at
DeadlineExceeded with nothing naming the discarded parameter.

Return the applied deadline together with a message when the configured
value is rejected, and record it as a BackupTimeoutInvalid Warning Event
on the BackupJob and RestoreJob (the reconcilers already hold a Recorder).
An honoured or unset value stays silent. TestKafkaRunDeadline now asserts
the warning fires for the rejected cases and not otherwise.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-14 17:55:48 +04:00
Andrey Kolkov caae13a60a docs(backups): drop the inaccurate NAMESPACE-override note from the demo runner
run-all.sh said NAMESPACE could be overridden through the environment, but
the numbered manifests pin namespace: tenant-root while only the waits read
$NAMESPACE, so an override would create resources in one namespace and wait
in another. The demo is tenant-root-specific anyway: the system bucket
credentials and the derived strategy live there. Say that instead.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-11 16:05:23 +04:00
Andrey Kolkov bd7d4abe84 fix(backups): size the Kafka metadata demo brokers at the chart default
The demo's source and to-copy Kafka fixtures requested resourcesPreset
"small", the deprecated legacy alias worth 1 CPU / 512Mi — half the Kafka
chart's own c1.small default (1 CPU / 1Gi). 512Mi is too little for the
broker: it crashlooped before the HelmRelease went Ready, so the demo
never reached the backup. Use c1.small on the kafka and zookeeper blocks
of both fixtures, matching the chart default and the in-tree kafka e2e
fixture that comes up cleanly.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-11 16:05:23 +04:00
Andrey Kolkov c3d1015df0 test(backups): pin the non-numeric recorded replication factor guard
The Kafka restore path rejects a recorded replication factor that is not
numeric, but only the partition-count guard had its own test. Dropping
both guards reddened that test, so the pair looked covered, while dropping
the replication-factor guard alone left the suite green. Add a case that
drives the create path (topic absent) with a valid recorded partition
count and a non-numeric replication factor, so the guard is pinned on its
own and a future edit that removes it fails loudly.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:28:22 +04:00
Andrey Kolkov c5a8a39bcd docs(backups): name the demo BackupClass in the Kafka flow headers
The run-all.sh flow header and the chainsaw harness header both said the
ad-hoc BackupJob runs against the platform cozy-default BackupClass. It
routes to this demo's kafka-metadata BackupClass instead, which maps Kafka
to the kafka-strategy-default strategy that run-all.sh derives from the
shipped cozy-default-kafka at run time. The numbered manifests already
described this; only the two summary headers overstated the one piece of
wiring the demo exists to exercise. Reword them to match.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 17825d4a32 fix(backups): declare the shell dialect in the Kafka backup script
The script opens with `set -euo pipefail` and no shebang, so shellcheck
reports SC2148 and cannot tell which dialect to lint against. Declare it
with a `# shellcheck shell=bash` directive as the first line: shellcheck
and pre-commit now lint it as bash, and the line is an inert comment once
the script is embedded into the Job's pod argument.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 2cb2b459fa fix(backups): reject a non-numeric recorded partition count on Kafka restore
The restore path validated only the shape parsed from the live cluster,
not the partition count and replication factor read from the stored
object. A non-numeric recorded partition count made `[ "${lp}" -lt
"${parts}" ]` and its `elif` exit 2; an `if` condition is exempt from
errexit, so both read as false, the topic was skipped, and the loop still
printed its per-topic success line - so the RestoreJob reached Succeeded
without ever touching the topic. A well-formed but wrong-content object,
which curl -f cannot catch, must fail loudly like the unrecognised-record
guard.

Validate the recorded parts and rf are numeric before the comparisons and
--create, with a bats case that reddens when the guard is dropped.

Also correct a stale to-copy example comment: the strategy reads the exact
object the source backup recorded in status.artifact.uri (injected as
ARTIFACT_URI), not a key derived from .Backup.ApplicationRef.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov cca3b08733 test(backups): pin the Kafka restore path and fail fast on a missing artifact
The restore half of the driver - the half
that mutates a live Kafka cluster - had no test reaching its happy path,
so three reversions of it left the suite green.

- Add restore reconcile tests mirroring the Rabbitmq driver this was
  copied from: ToCopy (pins the targetApplicationRef.Name merge - a
  to-copy restore must resolve the TARGET, not write the source's topics
  back into the source), NoRecordedArtifactURI, TargetClusterNotReady and
  TargetNotFound. Verified non-vacuous: neutralising the name merge reddens
  ToCopy, and reverting the fail-fast ordering below reddens
  NoRecordedArtifactURI.

- Fail fast on a Backup with no status.artifact.uri: move the check ahead
  of the readiness gate so it terminates immediately instead of waiting out
  the whole deadline and then reporting a misleading "timed out waiting for
  Kafka cluster to become Ready" (reconcileRabbitmqRestore checks it first;
  same order here).

- Add bats cases for the fail-closed guards that no test reddened: MODE
  validation, the s3:// URI prefix, cleanup treating 404 as success, an
  unparsable backup/live describe, a kafka-configs describe error, and an
  unrecognised record kind (plus a CFG_FAIL knob in the configs stub). The
  describe-parse guards abort via `set -euo pipefail` at the
  `X=$(describe_field ...)` assignment, so these pin the fail-closed
  behaviour; the explicit guards remain as belt-and-braces.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:53 +04:00
Andrey Kolkov 38d6c4cf58 fix(backups): bound Kafka metadata run at the Job layer, not a controller wall clock
A controller-side running-Job deadline was enforced in reconcileKafka /
reconcileKafkaRestore: at kafkaDefaultBackupDeadline from StartedAt the
BackupJob/RestoreJob was marked Failed. But the batch Job carried no
activeDeadlineSeconds and markBackupJobFailed only writes status, so the
Job kept running, finished, and its final `s3 PUT` uploaded the object —
while the terminal phase meant no Backup was ever created and cleanup
(keyed on Backup deletion) never ran. Every timed-out run thus orphaned a
uniquely-keyed object, and a cluster whose export legitimately exceeds the
deadline could never produce a usable backup. The shipped docs also
claimed the run "keeps the BackupJob Running ... rather than failing on a
wall clock", which the deadline made false.

Move the bound onto the Job as activeDeadlineSeconds (setKafkaJobDeadline):
Kubernetes fails an unschedulable or wedged Job as DeadlineExceeded and
kills its pod before the upload, so a timed-out run lands in the JobFailed
branch cleanly and leaves no object behind. Make it operator-tunable with
the `backupTimeout` BackupClass parameter (a Go duration), which
round-trips through the Backup so a restore is bounded by the same value;
the readiness-precondition wait honours it too. Correct the docs.

Also requeue on a transient broker-image List error instead of failing the
whole run: resolveKafkaBrokerImage now wraps a sentinel (errNoKafkaBroker)
so only a Ready cluster with no listable broker is terminal.

Tests: pin the deadline on the built Job, the backupTimeout override, a
JobFailed run marking the owner Failed, and the transient-List requeue.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 4925a84c26 fix(backups): pin the restore-side guards and bound a running Kafka Job
- delete_topic waited with `grep -qx` (BRE), so deleting `audit.events`
  matched the live `audit-events` and spun to its timeout — the e2e
  failed at step 25 on every run that reached it. Use `grep -qxF`.
- The two -k/--connect-to guards were written as `! grep`, which cannot
  fail under hack/cozytest.sh (it wraps each @test in `set -e` + a
  trailing `return 0`, and bash exempts a `!`-negated command from
  errexit). Rewrite them as positive `if grep ...; then return 1; fi`;
  verified the -k reinstatement now reddens the suite under cozytest.
- Cover the restore-side literal-topic guard that was unpinned at every
  level: a bats case runs the existing-topic branch with a colliding
  sibling and asserts the `--alter` argv carries `\Q…\E`, and run-all.sh
  now leaves `audit.events` live so the e2e exercises that branch instead
  of only `--create`. Add bats cases for the partition-shrink refusal, the
  `-f` upload guard, and `--exclude-internal`.
- Bound a running Job: reconcileKafka / reconcileKafkaRestore now fail the
  BackupJob/RestoreJob once kafkaDefaultBackupDeadline elapses in the
  Job-running branch (buildJobStrategyBatchJob sets no
  activeDeadlineSeconds), so an unschedulable pod no longer pins it in
  Running forever. Pinned by TestReconcileKafka_FailsWhenJobNeverCompletes.
- Fail closed on an unknown record kind and an unknown MODE, and validate
  the live partition/RF parsed on restore (an empty value silently skipped
  reconciliation). Correct the stale "image pinned in the strategy" comment
  in the example helpers and the chainsaw test (the controller now resolves
  the target broker's image as .ClientImage).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov 0d918e1f91 fix(backups): resolve Kafka backup client image from the target broker
The Kafka metadata strategy pinned kafkaBackupClientImage to an upstream
Strimzi tag with no producer: nothing stamped it and Renovate does not
manage helm-values, so it drifted from the broker version the operator
provisions. A dedicated small client image (the sibling curl-only drivers'
follow-up) is not viable here because the driver runs the Kafka Admin-API
CLI, which only the broker image ships.

Resolve the client image from a live broker pod of the target Strimzi Kafka
at reconcile time instead, so the CLI always matches the broker (any version
the operator picks, not only the chart default), and record it on the Backup
so cleanup can reuse it once the source cluster is gone. The chart no longer
pins the image; the strategy template renders it as the .ClientImage
placeholder the controller fills.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:23:52 +04:00
Andrey Kolkov ec26f1171d fix(backups): make Kafka metadata e2e reach an in-cluster S3 endpoint
The e2e drove run-all.sh against the shipped cozy-default-kafka strategy,
whose endpoint is the platform's EXTERNAL S3 ingress. In CI that host
(s3.example.org) is an unroutable placeholder with no issued cert, so the
backup Pod died on `curl: (6) Could not resolve host` and the BackupJob
went Failed — the chainsaw round-trip failed there, not in the driver.

Mirror the rabbitmq e2e model, but without duplicating the 150-line
driver script: run-all.sh now derives a demo strategy from the shipped
cozy-default-kafka at run time (jq-patch: override S3_ENDPOINT, and when
an endpoint override is given, mount the copied self-signed seaweedfs CA
and point CURL_CA_BUNDLE at it), applies a demo BackupClass, and routes
the BackupJob/Plan to it. The chainsaw run sets
S3_ENDPOINT=https://seaweedfs-s3.tenant-root.svc:8333 (the .svc FQDN the
seaweedfs cert SAN covers). No -k; TLS is verified against the CA. With
no override (a real cluster whose external endpoint resolves and uses a
public CA) the shipped endpoint is reused and no CA is mounted, since
CURL_CA_BUNDLE replaces the system bundle rather than extending it.

The shipped cozy-default-kafka strategy is untouched. jq derivation
verified on a sample in both CA modes; example scripts pass bash -n.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:34 +04:00
Andrey Kolkov 06e16d5f47 fix(backups): make the Kafka bats run under cozytest.sh (CI runner)
The suite was written for upstream bats and passed there, but CI runs
hack/cozytest.sh, which knows only @test + bash: it never calls setup()
and provides no $BATS_TEST_TMPDIR / run / $status / $output. So setup()
never ran and the first test aborted with "STATE: parameter not set"
under set -u. Move the stub setup into an init_stubs helper each test
calls, drop the bats builtins for plain exit-code and $(...) assertions,
and mktemp per test instead of $BATS_TEST_TMPDIR. Verified with
hack/cozytest.sh (11/11), the runner make bats-unit-tests uses.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:34 +04:00
Andrey Kolkov aaadc70a08 docs(backups): drop a decaying line-count from the Kafka bats header
The comment cited "the 160-line driver script"; a count about the file
decays on the next edit and nothing pins it. State the invariant instead.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:33 +04:00
Andrey Kolkov ebb3118694 fix(backups): key Kafka --connect-to off the request host in both S3 modes
The port workaround built --connect-to's HOST1 from ${host}, but the
virtual-hosted branch (forcePathStyle:false) sends to ${bucket}.${host}.
curl matches HOST1 against the request host, so on a virtual-hosted AND
non-default-port endpoint the redirect never fired and the request hit the
wrong port. Derive HOST1 from the host the URL actually uses; the real
connection target stays ${host}:${port}. Path-style is unaffected. Add a
bats case for the virtual-hosted ported mode, which the suite did not cover.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:32 +04:00
Andrey Kolkov 14c6af122e fix(backups): restore Kafka S3 port workaround and hold the script with a test
- Regression fix: an earlier change dropped the curl --connect-to port workaround along
  with -k, but curl 7.76.1 (the image's) 403s on ANY ported endpoint,
  which is the shape backupStorage.endpoint ships (:8333) and what
  provisionBucket:false and s3.ingress.enabled:false produce. Restore
  --connect-to (sign a portless URL, redirect to the real port), keeping
  -k off so TLS is still verified. IPv6-safe host/port split, path prefix
  preserved. The port-less ingress passed; the ported case is the one that regressed.
- Correct the S3 envelope claim in values.yaml and backup-classes.md: the
  403 is decided by the port, not a body-hash the backend recomputes, and
  --connect-to handles it on SeaweedFS and MinIO alike.
- Hold the script fixes at the layer the contract lives at. Externalise
  the driver to files/kafka-backup.sh (embedded verbatim via .Files.Get,
  render byte-identical) and add hack/kafka-backup-script.bats, which
  EXECUTES it against stubbed kafka CLIs + curl and asserts behaviour: \Q
  literal match on the colliding pair, --connect-to with no -k on a ported
  endpoint, fail-closed --list, comma-value bracket grouping, RF-divergence
  abort, and the cleanup delete's HTTP handling. Not a template grep.
- Pin the Backup-delete dispatch and the readiness deadline in Go:
  TestCleanupOnDelete_Kafka_RoutesToKafkaCleanup and
  TestReconcileKafka_FailsAfterReadinessDeadline.
- Docs: config restore is additive (no --delete-config); export time
  scales with topic count and the Job has no wall clock (as its siblings).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:32 +04:00
Andrey Kolkov ed3cb34cbf fix(backups): adopt-guard Kafka Backup artifact and correct URI-template doc
- createKafkaBackupArtifact now rejects adopting a same-named Backup that
  describes a different application, so a retained Backup reused for a
  different app of the same name is not reported as this run's success.
  Mirrors the RabbitMQ driver's guard; pinned by a Go test.
- ArtifactURITemplate godoc said empty leaves status.artifact unset, but
  the driver hard-fails an empty template. Reword to state it is required,
  and regenerate the CRD description to match.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:07 +04:00
Andrey Kolkov 0270650237 fix(backups): harden Kafka metadata S3 handling and own the artifact
- Drop `-k`: mirror the RabbitMQ S3 layer (ARTIFACT_URI as the single
  source of truth, forcePathStyle honoured, whole base URL kept) so
  curl --aws-sigv4 verifies TLS against the ACME endpoint the helper
  returns. The derived endpoint is default-port, so the curl 7.76.1
  --connect-to port workaround is gone too; this also fixes the IPv6
  and path-prefix mangling in the old host-splitting.
- Own the artifact: a cleanup mode in the script plus cleanupKafkaBackup
  wired into BackupReconciler.cleanupOnDelete delete a deleted Backup's
  object (best-effort, waited on) instead of orphaning it, matching the
  RabbitMQ precedent. Kafka has the identical own-your-object property.
- Restore probes existence with --list (fail closed), so an unreachable
  broker aborts instead of being misread as "topic absent" and created.
- Hold the fixes with tests at the right level: a colliding topic pair
  (audit.events / audit-events, distinct partition counts) in the e2e so
  a revert of the \Q...\E literal-match breaks the round-trip, and Go
  tests pinning the scope stamp and the cleanup Job.
- Correct the docs/comments: endpoint-shape row, the "cached on nodes"
  claim, and the narrower curl 7.76.1 S3 envelope (empty-body payload
  hash) that makes a small modern-curl client image the right follow-up.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:22:06 +04:00
Andrey Kolkov 537aadf389 fix(backups): fix Kafka backup S3 signing and literal topic names
Reproduced end-to-end against a real Kafka 3.9.1 broker:

- [CRITICAL] The Strimzi image ships curl 7.76.1, whose --aws-sigv4
  omits a non-default port from the SigV4 canonical Host while sending it
  in the Host header, so every S3 PUT/GET 403s and the driver never
  uploads or downloads. Sign/send the bare host and redirect the
  connection to the real port with --connect-to (the same fix the
  topic-data example already carries), and accept the in-cluster
  self-signed cert with -k.
- [CRITICAL] kafka-topics --topic is a Java regex, not a literal: a
  backup entry for "audit.events" also matched a live "audit-events", so
  restore repartitioned an unrelated topic irreversibly and backup
  recorded the wrong partition/replication values. Wrap the name in
  \Q...\E on every filter call (--describe on both paths, --alter on
  restore) so it matches exactly one topic; --create and
  kafka-configs --entity-name are already literal. Applied in the example
  helpers too.
- [MAJOR] cozy-default's Kafka route looked like a data backup at
  runtime. Stamp scope=topic-metadata on the produced Backup's
  DriverMetadata and say "topic definitions and configs only; no message
  data" in the Ready condition, so kubectl describe backup is honest.
- [MAJOR] Add reconcile tests for the two Go behaviours a mutation could
  silently drop: the Strimzi-Ready precondition on backup (BackupJob
  stays out of Succeeded and creates no Job) and the applicationRef Kind
  gate on the restore target.
- [MINOR] Fail closed on the config-capture grep (exit != 1 aborts) and
  clarify the image pin has no automated producer.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:25 +04:00
Andrey Kolkov 405eb25095 fix(backups): harden Kafka metadata backup
- Fail closed on topic listing: set -o pipefail, capture kafka-topics
  --list --exclude-internal exit separately, and abort on failure so an
  unreachable broker can no longer produce an empty backup reported as
  Succeeded. An empty list after a successful call stays legitimate.
- Check each kafka-topics --describe exit and validate the parsed
  partition count / replication factor before writing a record, so a
  timed-out describe cannot write a corrupt orders\t\t\t line.
- Store topic configs one per line with key and value in separate
  tab-delimited fields, sourced from kafka-configs --describe (one config
  per line), so a value containing a comma (cleanup.policy=compact,delete)
  round-trips; on restore each config is applied with the value bracketed
  only when it contains a comma.
- Detect topic divergence on restore instead of a silent --if-not-exists
  no-op: grow partitions with --alter when the backup asks for more, and
  fail loudly (naming the topic and both values) when it asks for fewer
  partitions or a different replication factor. A restore that cannot
  reach the recorded state no longer reports Succeeded.
- Add KafkaStrategyKind to the RestoreJob cleanup-dispatch no-op group so
  deleting a Kafka RestoreJob does not run a pointless Velero cleanup; add
  a regression test alongside the siblings, plus a reconcile Kind-gate
  test and a rendering test that actually renders the strategy template.
- Wire the kafka-metadata Chainsaw suite into both selectors
  (select-install suite_to_source, select-e2e src_to_suites) so editing
  the kafka app or the suite selects it and the bats unit tests pass.
- Pin the client image to the operator's default broker version
  (kafka 3.9.1) with a digest, and align the example default; fix the
  values comment. Point the RabbitMQ comment references (which shipped in
  the CRD description) at the Job strategy actually mirrored, and trim the
  in-pod comments that rendered into the manifest.
- Document the Strimzi Topic Operator caveat: for a topic owned by a
  KafkaTopic CR the operator reconciles its config from the CR, so an
  in-place config restore is durable only for out-of-band topics.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:21:24 +04:00
Andrey Kolkov a410a3f48b feat(backups): Kafka topic-metadata backup via a dedicated strategy driver
Add a strategy.backups.cozystack.io/Kafka driver that backs up the topic
metadata of a Cozystack-managed Kafka application — partition counts,
replication factors and non-default topic configs — through the Kafka
Admin API, and restores it with kafka-topics --create + kafka-configs
--alter. Modelled on the Job/Rabbitmq strategies: the driver renders a
one-shot batch Job, gates on the Strimzi Kafka cluster reporting Ready,
and keys each backup by .BackupName so runs do not overwrite each other.

Scope is deliberately narrow: message payloads, consumer-group offsets,
ACLs, quotas and KafkaUsers are out of scope, and there is no PITR.
Restored topics are created directly (unmanaged on the target) — the
accepted tradeoff of an Admin-API-only driver. Both restore flows are
implemented: in-place recreates dropped topics into the source, to-copy
applies the source's topics onto a freshly-bootstrapped empty cluster.

Wire it into cozy-default (strategy-kafka-default.yaml gated on the
bucket name, a BackupClass binding, read RBAC on kafka.strimzi.io/kafkas
for the readiness gate, and the Strimzi Kafka client image), document it
in backup-classes.md, and add an honest round-trip example plus Chainsaw
e2e (examples/backups/kafka-metadata, kafka-3-metadata-roundtrip) that
proves a per-run retention.ms sentinel survives both restores.

Unit tests cover the driver's pure helpers (application-ref validation,
readiness verdict, parameter round-trip, per-run artifact-URI keying).

Not yet verified on a live cluster.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-10 22:20:27 +04:00
scooby87 f7cb9e7854 fix(postgres): follow-up review remarks from #3954 (KEDA autoscaling) (#4188)
Follow-up to #3954 (postgres read-replica autoscaling via KEDA), closing
the non-blocking review remarks tracked in #4187. Each change is small,
isolated, and covered by a red→green test where it is a behaviour
change.

## Changes

- **fix(postgres): pin `job="kube-state-metrics"` on the autoscaler
read-load query.** The trigger query joined on `kube_pod_labels` without
the job pin the sibling `DatabaseAutoscaler*` alerts already carry; a
duplicate `kube_pod_labels` series from another exporter would multiply
the join. Pinned on every selector; unittest asserts it in both the base
and lag-brake query shapes.
- **fix(cozy-lib): reject a negative `minReplicaCount` in
`cozy-lib.keda.scaledObject`.** Only `max >= min` was checked, so a
mixed-sign pair (`min -5, max -3`) passed and rendered a nonsensical
floor. Now failed closed before the ordering check, with a unit case.
Also added coverage for the previously-untested `annotations` parameter.
- **docs(postgres): correct the KEDA-missing guard rationale and note
the gradual client-side rollout.** The guard comment still described the
replaced drop-instances design; corrected to the seed design (a skipped
ScaledObject leaves a fixed-size cluster with no HPA, not a collapse to
1). The cozyrds apply-strategy annotation now notes that the client-side
switch rolls out per-app as each Application is next written through the
aggregated API, not fleet-wide at upgrade.

## Testing

`helm unittest` green: postgres 66/66, cozy-lib-tests 45/45, keda 12/12.
Both behaviour changes are mutation-verified (reverting the fix reddens
the suite). No generated artifacts affected (no
`values.yaml`/`values.schema.json`/`Chart.yaml`/`README.md` changes).

## Release note

```release-note
NONE
```

Closes #4187


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved PostgreSQL autoscaling metrics by ensuring pod data is
sourced from the correct metrics exporter, preventing duplicate data
from distorting scaling decisions.
  - Added validation to reject negative minimum replica counts.
- Preserved caller-provided annotations on generated autoscaling
resources.

- **Documentation**
- Clarified PostgreSQL autoscaling fallback behavior when the
autoscaling controller is unavailable.
- Documented the gradual transition of existing PostgreSQL resources to
the updated apply behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-10 19:32:24 +03:00
Aleksei Sviridkin 7c739a14fa fix(cdi): omit the CDI clone strategy override when unset (#4138)
## What this PR does

`cdi-cr.yaml` rendered `cloneStrategyOverride` unconditionally, so there
was no way to let CDI pick a clone strategy per StorageProfile. The CRD
enum is `copy`/`snapshot`/`csi-clone` with no empty member, so neither
obvious workaround got there: an empty value rendered
`cloneStrategyOverride: ""`, which that enum does not accept, and a null
one rendered a bare key. Both checked against `main` with `helm
template` rather than assumed.

Now the key is guarded on a non-empty value, same as
`uploadProxyURLOverride` just below it in the same file. With the field
absent, CDI reads the target StorageClass's StorageProfile and uses its
clone strategy.

Default stays `csi-clone`. Rendered at the default and diffed against
`main`, the output is byte-identical, so nothing already deployed moves.

### Tests

`tests/clone_strategy_test.yaml` pins both directions: absent when the
value is empty, absent when it is null, `csi-clone` on the chart
default, `snapshot` through unchanged. It sits alongside the existing
`importer-resources` and `standalone_render_test` suites, with its own
file and suite name and no shared test names; `make test` runs all
three.

I checked the suite can fail rather than assuming it works. Reverting
the guard reddens the two absent-field cases, deleting the field
outright reddens the default and passthrough cases, hardcoding the value
reddens the passthrough case, and rendering a bare key reddens all four.
The two sibling suites stay green through every one of those, so the
reds are attributable to this suite.

The null case is pinned as an input rather than argued from `with`
treating null and empty alike. It cannot pass vacuously: flipping its
assertion to expect the default returns `unknown path`, not a
`csi-clone` mismatch, which is what shows the null actually reaches the
template instead of the assertion passing on a value that never arrived.

### Known gaps

`copy` is accepted by the enum but no test drives it, so the suite does
not pin that it reaches the CR. Concretely: wrapping the guard in `{{-
if ne . "copy" }}`, which silently drops that one strategy, still passes
all nine tests. Named here rather than fixed so the boundary is visible
to the next reader instead of being rediscovered.

### Scope

Context: https://github.com/cozystack/cozystack/issues/2908. This is the
first of the three directions that issue lists, so it does not close it
and carries no `Closes`. The default is deliberately left at
`csi-clone`, since whether that is the right default is the second
direction and a separate decision. The third, `vm-disk` warning on a
cross-StorageClass request, is untouched. An operator who has not set
`cloneStrategyOverride: ""` sees the reported symptom exactly as before.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. It is one guarded key in a
`packages/system` chart, a values comment, and a test suite: no package
added or renamed, no `values.schema.json` (this chart has none), no
`ApplicationDefinition`, no namespace, nothing under `hack/`, no
platform values key. The `ccp` entry covers `hack/package.mk` and
make-target behaviour, and this touches neither. Nothing in the map
reaches it.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cdi): `cloneStrategyOverride` in the `kubevirt-cdi` package can now be left empty, which omits the field and lets CDI pick a clone strategy per StorageProfile. The default is unchanged at `csi-clone`.
```
2026-09-10 19:22:21 +04:00
Aleksei Sviridkin 4b6b1dc2da fix(vm-instance): let complete resources replace the instance type (#4109)
## What this PR does

Closes #2909.

The matcher keyed off `instanceType`, the domain sizing keyed off
`resources`, and neither side looked at the other, so the chart could
render both. KubeVirt refuses that, and the two halves of its rule are
different shapes: `validateCPU` conflicts on `domain.cpu.cores` and
`domain.cpu.sockets` only when they are non-zero, while `validateMemory`
conflicts on `domain.resources.requests.memory` whenever the key is
present. `values.yaml` ships `instanceType: "u1.medium"`, so clearing
the field in the dashboard just falls back to that default. Fill in cpu,
sockets and memory after clearing it and you get a VM the webhook
rejects, naming fields the form gives you no way to clear.

Resources that size the VM completely now suppress the matcher and size
it directly. `packages/apps/kubernetes-nodes` already does this for its
worker VMs. Beside an instance type, sizing that is only partial stops
the render, naming what sized the VM and what did not; with no instance
type in play there is nothing to conflict with, so that half renders as
it does today. Letting resources win unconditionally would silently drop
the instance type for a VM that sets memory alone, which is a different
bug in the same place.

Completeness is decided from what `domainResources` emitted, never from
which keys the user set, and all three fields go through that one
reading. The two readings come apart in more than one place.
`resources.cpu` on its own emits nothing, so it conflicts with nothing
and goes on rendering a plain instancetype-sized VM. cpu and sockets
pass through `int64`, which yields 0 for a quantity the schema accepts,
so `cpu: 500m` beside a whole sockets and a memory would read as
complete from the values, drop the matcher, and hand KubeVirt a `cores:
0` the user never asked for. And a `resources` block nulled outright
leaves `.Values.resources` nil rather than the chart's `{}`, so a raw
field read is a render-time nil pointer where the emitted object is
merely empty.

The existence check and the pre-upgrade update hook read the same
effective value. Keying the `lookup` on the raw one would fail a
resource-sized render over an instance type it never references, which
is the same confusing error this fixes. Reading `.Values.instanceType`
in the hook would see the leftover value, skip the removal patch, and
leave `spec.instancetype.revisionName` on a VM Helm had just given
explicit domain resources.

### What changes in the rendered output

| `instanceType` | `resources` | Before | After |
| --- | --- | --- | --- |
| set | unset | matcher only | unchanged |
| unset | cpu, sockets and memory | domain cpu and resources | unchanged
|
| set | cpu, sockets and memory | matcher plus domain sizing, refused by
KubeVirt | domain sizing, no matcher |
| set | any subset with memory, or cpu with sockets | matcher plus a
conflicting domain field, refused by KubeVirt | render stops, message
names both sides |
| set | cpu alone, or sockets alone | matcher only | unchanged |
| set | cpu and sockets that both truncate to zero | matcher plus
`cores: 0` | unchanged |
| unset | anything short of all three | already refused by the existing
check | unchanged |

The rows that stay unchanged are deliberate.
`virtual-machine.domainResources` writes `domain.cpu` only when cpu and
sockets arrive together, so cpu alone emits no override at all. A
`cores: 0` beside a matcher is not a conflict either, because
`validateCPU` guards on non-zero, so KubeVirt admits it and the instance
type sizes the VM. Both shapes render today and go on rendering.
`cpuModel` is left alone for a related reason: it writes
`domain.cpu.model`, which KubeVirt flags only when the instance type
declares a model of its own, and no instance type shipped in
`packages/system/kubevirt-instancetypes` declares one.

### Why no working release breaks

Every shape that now fails at render was already being rejected by the
KubeVirt validating webhook at admission, so it could not have produced
a running VM in the first place. The failure moves from an opaque
admission error to a render error that names the fields. Nothing that
worked stops working.

The unchanged rows above are measured. Each non-failing shape in the
table, plus firmware on and off, an explicitly nulled `resources` block
and the three update-hook branches, was rendered against a mocked
cluster on both sides and compared document by document. Across 14 value
shapes and 78 rendered documents, 77 come out byte-identical. The one
that differs is the row the table marks as changed, which is also what
the harness was checked against first, so it is known to be able to
fail.

### Live-VM impact

This changes a rendered VM spec, so an upgrade can resize or restart a
running VM. Adding a complete `resources` block to a VM currently sized
by an instance type drops its matcher, and the pre-upgrade hook patches
`spec.instancetype` off the live object and writes the domain resources
in its place. Cozystack runs KubeVirt with `vmRolloutStrategy:
LiveUpdate` (`packages/system/kubevirt/templates/kubevirt-cr.yaml:125`),
so KubeVirt applies what it can to the running VMI and marks the rest
`RestartRequired`; I have not measured which half a given resize lands
in, so plan the upgrade as if it restarts.

Dropping the matcher drops the whole instance type, sizing and
everything else it carries. Of the 69 shipped cluster instancetypes, 36
declare `dedicatedCPUPlacement`, 36 declare `hugepages` and 26 declare
both. 20 declare NUMA passthrough, 8 declare `realtime`. A VM moved from
`rt1.medium` to explicit resources loses those characteristics along
with the type. Before this change that combination was rejected
outright, so the VM never existed; now it is admitted, and the field
help says so.

### Tests

`tests/instancetype_resources_test.yaml` covers matcher only, resources
only, both complete, each failing partial shape, the shapes that emit
nothing, both cpu and sockets truncating to zero, `cpuModel` beside a
matcher and beside complete resources, a stale instance type with and
without complete resources, an explicit `instanceType: null`, an
explicitly nulled `resources` block, and every branch of the update
hook's instancetype selector, in both directions plus the two no-op
cases. 24 cases in this package, 2127 across `make helm-unit-tests`, all
green. Every case pinning new behaviour was checked against a reverted
template to confirm it goes red; the rest pin behaviour this change
deliberately leaves alone.

A ten-mutation ladder over the helper, `vm.yaml` and the update hook
leaves no survivor. The memory branch took the most work. Swapping its
emitted read for a raw `.Values.resources.memory` looks like an
equivalent mutant, since `domainResources` emits
`resources.requests.memory` under the same truthiness condition. It is
not: a `resources` block nulled outright leaves `.Values.resources` nil,
and the raw read dereferences it. The case pinning that shape kills the
mutant.

The update-hook cases assert the `kubectl patch` command the Job will
run, which is as far as a render-level test reaches. They do not show
that KubeVirt accepts the resulting object, that helm-controller
converges on a live VM whose `spec.instancetype` also carries a
`revisionName` written by virt-controller, or which half of a resize
lands as a live update rather than `RestartRequired`. That needs a
cluster.

The `dig` calls survive the shapes that reach them. `values.schema.json`
rejects a non-object `resources` before render; Helm coalescing turns a
nulled one into a nil `.Values.resources`, which digs to an empty
object; and for any other non-map intermediate, current Helm turns the
underlying sprig type assertion into a render error rather than a panic.

### Interaction with the firmware block

`firmware` (UEFI/BIOS selection, Secure Boot, persistent EFI NVRAM)
landed in this chart in the meantime, and it does not touch the sizing
question. `domainResources` does not read it, so it takes no part in the
completeness decision. KubeVirt's instancetype spec carries neither
firmware nor `domain.features`, so nothing an instance type declares can
conflict with what the firmware block renders. The boot mode a VM
inherits when the block is empty comes from the preference, and the
preference is not what gets dropped when complete resources replace the
matcher. Both firmware states are covered on each side by the
byte-identity comparison above.

`README.md`, `values.schema.json`,
`api/apps/v1alpha1/vminstance/types.go` and the ApplicationDefinition
all come out of `make generate` in the package, not out of a hand merge.
Re-running it on the result produces no further diff, and the root `make
generate` reports no drift on top of it.

### On the ticket's second proposal

#2909 also asks whether the non-empty `u1.medium` default belongs in the
chart at all. That requirement is met, and not by removing the default.
The default was a problem because a schema-generated form gave no way to
say "no instance type": clearing the field falls back to `u1.medium`.
Filling cpu, sockets and memory now says it directly, so the obstacle
the ticket names is gone while the default stays. Dropping the default
on top of that would change the rendered spec of every release that
never set the field, with nothing left to gain.

### Named improvements, not done here

The failure message names the fields that did not size the VM, computed
from what was emitted, so for `cpu: 4` with no `sockets` it names
`resources.cpu` even though the user set it. The message states the pair
rule that explains this, but naming the actually-missing field first
would read better. The two tests that pin `domain.cpu.cores: 0` document
the pre-existing `int64` truncation in prose rather than through a
`FIXME` reference, so a later reader has to read the comment to tell a
contract from a bug we chose not to fix. Both are wording; neither
changes behaviour.

`resources.cpu`, `sockets` and `memory` are checked for shape and not
for value, so `memory: "0"` and `-1Gi` pass the schema and fail further
down. Both forms fail before and after this change and only the place of
the failure moves, so that belongs with
[#4104](https://github.com/cozystack/cozystack/issues/4104) rather than
here.

The new template comments run to about 46 lines against about 28 lines
of logic. All of it is why rather than what and all of it is accurate,
but the helper's third paragraph restates the commit body, and two
sentences would carry that weight. The seven-line block in `vm.yaml` is
the one worth keeping: swapping that gate to the effective value
survives the entire suite, so what the block records is an equivalence
no test can pin, and trimming it would delete the only record of it.

The `instanceType` description runs to about 370 characters and reaches
the dashboard as tooltip help, through `values.schema.json` and the
ApplicationDefinition. That is long for a tooltip. It is accurate as it
stands, and rewriting accurate text for length is a fresh unmeasured
claim in the surface with the most readers, so it is not done here: the
place for the detail about what a dropped instance type takes with it is
the chart README, leaving the field help carrying the precedence rule
alone.

Choosing a `cx1.*` type and then giving it complete resources drops what
the type guaranteed along with the type, and nothing says so at runtime.
That is a consequence of the precedence rule rather than a defect in it,
and the field help states it. A runtime signal would need the chart to
compare what the type declares against what `resources` sets, which is
its own task.

Defaulting `sockets` to 1 would let a user size a VM from cpu and memory
alone and would remove the commonest way to land in the partial case. It
is more than the ticket asked for and it changes the rendered spec of
stored releases that set cpu and memory without sockets, so it needs its
own call on compatibility.

### Not changed

Two pre-existing quirks stay as they are, because both reach past the
conflict this fixes. `resources` fields are read for truthiness, so an
explicit `0` reads as unset, in `domainResources` and in the existing
"Either instanceType or resources" check alike. And with no instance
type in play, a fractional `resources.cpu` still truncates to `cores: 0`
and renders a VM sized by nothing, with no error. The new guard covers
only the half where a matcher is present. The tests pin both as they
stand, which is what keeps working releases rendering.

### Screenshots

No dashboard code is touched. The only user-visible surface is two field
descriptions the dashboard renders as help text, regenerated into
`values.schema.json` and the ApplicationDefinition. `instanceType` now
reads "Ignored when `resources` sizes the VM, which needs whole-number
cpu and sockets plus memory; the whole type is dropped then, so any
dedicated CPU placement, hugepages, NUMA or realtime settings it carries
go with it. While this field is set, a `resources` block that sizes only
part of the VM is rejected outright instead of ignored, so complete it
or clear it." `resources` states the same rule from its own side.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

I walked the trigger map against the diff and no listed trigger fires:
no package added, renamed or removed; no `values.schema.json` field
added, renamed or removed; no default changed; nothing under `hack/`
moved; `templates/service.yaml` and its `wholeIP` and `allowICMP`
annotations untouched. Two repositories do carry prose this makes
incomplete. I left every box empty rather than push follow-ups nobody
agreed to, so a maintainer decides.

`cozystack/website`: `content/en/docs/next/virtualization/resources.md`
opens with "Each virtual machine has these two configuration settings"
and lists only `instanceType` and `instanceProfile`. The page is
hand-written, so the release-time bot that regenerates app reference
pages from `README.md` will not reach it.

`cozystack/terraform-provider-cozystack`:
`internal/provider/vminstance_schema.go` describes `instance_type` as
"Virtual machine instance type." Its `kubernetes_schema.go` and
`kubernetes_nodes_schema.go` already spell out the equivalent precedence
rule for their own charts. No schema field or default changed here, so
its new-field test guard has nothing to catch.

### Release note

```release-note
fix(vm-instance): a complete `resources` block (whole-number cpu and sockets, plus memory) now sizes the VM directly and suppresses the `instanceType` matcher, instead of rendering both and having KubeVirt refuse the VM. Beside an `instanceType`, a `resources` block that sizes the VM only in part now stops the render with a message naming both sides, where it previously produced a VM the admission webhook rejected. Note that suppressing the matcher drops the whole instance type, including any dedicated CPU placement, hugepages, NUMA or realtime settings it carries. A release whose values already carry both converts on the next platform upgrade rather than at a moment the operator picks: the pre-upgrade hook strips `spec.instancetype` off the live VM and writes the domain sizing in its place, so plan that upgrade as if the VM restarts.
```
2026-09-10 19:21:59 +04:00
Aleksei Sviridkin 1f04e31ed3 feat(backups): add Redis backup/restore strategy driver (#3992)
## What this PR does

Adds backup/restore support for the `Redis` app (spotahome
`RedisFailover`) via a dedicated `strategy.backups.cozystack.io/Redis`
strategy controller. The spotahome operator ships no backup CRD, so the
driver follows the generic-script-Job pattern (a specialisation of the
Job/Altinity drivers): it runs a one-shot Job that dumps an RDB snapshot
from the current master to the system bucket and replays it on restore.

- **Backup**: discover the current master via the operator's sentinel
Service (`rfs-<name>`, master `mymaster`) → `redis-cli --rdb` → `curl
--aws-sigv4` PUT to `s3://<bucket>/<namespace>/<app>/dump.rdb`.
- **Restore**: GET the object → convert RDB → RESP (`hdt3213/rdb`) →
`FLUSHALL` + `redis-cli --pipe` into the target's master. Both
**in-place** and **to-copy** are implemented; a to-copy restore reads
the source's prefix via `.Backup.ApplicationRef.Name` while writing into
a differently-named target.
- **PITR**: not supported — RDB is a discrete snapshot with no
continuous log to replay to an arbitrary time (the etcd-like case).

The driver is a specialisation of the Job/Altinity script-Job flow with
a `Redis` applicationRef gate and an app-readiness precondition
(surfaces a precise `Ready=False` instead of a silent hang; blocks only
on an explicit `Ready=False` so a missing condition never wedges
backups). Strategy parameters round-trip through the Backup's
`DriverMetadata`; there is no `underlyingResources` snapshot (the Job
pattern re-renders on restore).

### System bucket

Coordinates and credentials reach the Job as env `secretKeyRef` from the
projected `cozy-backups-creds` (the ClickHouse/Altinity delivery model),
so **the app chart is untouched** — no `backup.useSystemBucket`, no new
RBAC (`batch/jobs` and `apps.cozystack.io/*` are already granted). The
bare-host `endpoint` gets `https://` prepended when unscheme'd; against
a self-signed endpoint the strategy verifies with the CA from
`backupStorage.endpointCASecretName` when set, otherwise against the
image's system CA store; skip-verify is a separate explicit
`backupStorage.insecureSkipTLSVerify` opt-in, never a fallback. The
`cozy-default-redis` strategy is gated on `$bucketName` (CRD-ordering)
and bound in the `cozy-default` BackupClass.

### Changes

- `api/backups/strategy/v1alpha1/redis_types.go` + generated CRD and
deepcopy.
- `internal/backupcontroller/redisstrategy_controller.go` (+ unit tests)
wired into both dispatch switches, `supportedBackupStrategyKinds`, and
the no-op restore-cleanup case.
- `cozy-default-redis` strategy template + `cozy-default` BackupClass
route + helm rendering-gate coverage.
- `images/redis-backup` (redis-cli + curl + RDB→RESP converter) +
Makefile target + `backupStrategyController.redisBackupClientImage`.
- `examples/backups/redis/` — an honest marker round-trip for both
restore flows.

### Validation

`go build`/`vet`, `go test ./internal/backupcontroller/...
./api/backups/...`, `make generate` (clean), and `helm unittest` (10/10)
all pass. Not yet cluster-verified.

## Downstream

None.

```release-note
Add backup and restore support for the Redis application (spotahome RedisFailover) via a new Redis backup strategy, wired into the cozy-default BackupClass. Supports in-place and to-copy restore; PITR is not supported.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Redis backup and restore support using RDB snapshots in
S3-compatible storage.
  * Supports restoring data in place or to a separate Redis instance.
* Added configuration for Redis backup images, custom HTTPS certificate
authorities, and TLS verification.
* Added a complete Redis backup and restore demonstration with automated
setup, verification, and cleanup.

* **Documentation**
* Documented Redis storage, credentials, endpoints, TLS, and restore
behavior.
  * Added administrator and tenant scenario guides.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-10 19:17:26 +04:00
Alexey Artamonov 9033d41bc0 test(postgres): assert the job pin on each autoscaler selector, not once for the query
The pin check matched job="kube-state-metrics" anywhere in the trigger query, so
one pinned selector satisfied it no matter what the other three did: unpinning
$joinReplica, $idleFloor or $joinCluster individually left the suite green while
the join went back to matching any exporter's series. Anchor one assertion per
selector on the label that identifies its site so a single dropped pin fails.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-10 15:31:31 +03:00
Alexey Artamonov 0d8b249b15 fix(cozy-lib): guard fractional-negative minReplicaCount and cover the pause-merge collision
The negative-floor guard read $min, already int .minReplicaCount, and int
truncates toward zero, so a fractional negative such as -0.5 became 0 and slipped
past into exactly the scale-to-zero-capable ScaledObject the type check above
exists to prevent. Test the untruncated value instead.

The annotations test only set an extra key with no pause, so it exercised the
pass-through branch and never the merge where the pause pair is the destination
and must win a key collision. Pass a colliding paused-scale-in="false" alongside a
distinct key and assert the pause value overrides it on the rendered object.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-10 15:31:24 +03:00
Aleksei Sviridkin e89617d754 fix(opensearch-operator): align DNS_BASE with the platform cluster domain (#4185)
## What this PR does

The opensearch-operator resolves every in-cluster name it dials as
`<svc>.<ns>.svc.<DNS_BASE>`, and nothing set `DNS_BASE`, so it stayed on
the vendored chart default `cluster.local` while the platform runs on a
configurable domain. On a cluster that is not on `cluster.local` the
securityconfig bootstrap job waits forever on a name that never
resolves, the security configuration is never applied, and OpenSearch
nodes stay unready. That is issue #4073.

The platform now carries the domain to the operator in the
`cozystack-values` Secret, under the subchart key the vendored chart
already reads:

```yaml
opensearch-operator:
  manager:
    dnsBase: "cozy.local"
```

`charts/opensearch-operator` reads that as `.Values.manager.dnsBase` and
hands it to the manager as `DNS_BASE`, so nothing under `charts/`
changes and the package needs no patch. The value renders from
`networking.clusterDomain` rather than a literal, so a cluster that
keeps `cluster.local` keeps working, and a package rendered without the
Secret keeps the upstream `cluster.local` default. This is the route the
KEDA cluster domain already travels through the same Secret.

The whole change is one key in
`packages/core/platform/templates/apps.yaml` plus two test suites: one
pinning that `networking.clusterDomain` reaches the Secret under that
key, one pinning that the key reaches the Deployment env in the package.

### Why this is not the `global` route

Earlier revisions of this branch, and #3357 before it, emitted the
domain as `global.clusterDomain` and patched the vendored template to
read it. `global` is the one key Helm copies into every subchart, which
is why it looked right. That is also the problem. The `cozystack-values`
Secret is `valuesFrom` on every HelmRelease the operator builds unless
the PackageSource opts out with
`operator.cozystack.io/skip-cozystack-values`
(`internal/operator/package_reconciler.go:265-273`), so a new key under
`global` lands in the values of nearly every release on the cluster.

One of them broke on it. The vendored smee chart in
`packages/system/bootbox` gates on the truthiness of the whole `global`
map and then indexes `global.rbac`, so a single unrelated key under
`global` ends its render. This still reproduces on `main`:

```console
$ helm template bootbox packages/system/bootbox
# exit 0
$ helm template bootbox packages/system/bootbox --set global.clusterDomain=cozy.local
Error: cozy-bootbox/charts/smee/templates/role.yaml:4:32
  executing "cozy-bootbox/charts/smee/templates/role.yaml" at <.Values.global.rbac.type>:
    nil pointer evaluating interface {}.type
```

The subchart key reaches the operator with no patch, no change to any
other release, and nothing to harden in bootbox. I measured that rather
than reasoning about it: rendering all 170 charts under `packages/*/*`
plain and with `--set opensearch-operator.manager.dnsBase=cozy.local`,
exactly one chart's output changes, opensearch-operator, by the one
`DNS_BASE` line this PR is about. Nine other charts differ between the
two runs by exactly as much as two identical runs of the same command
differ, which is the random material Helm regenerates per render, and
thirty-five fail in both modes for reasons unrelated to the flag, with
byte-identical errors either way.

The smee hardening is still worth having, because the next key anyone
puts under `global` hits the same gate, and that gate is still on the
default branch of tinkerbell/charts. It is not carried here. It is
available as a separate change against its own package.

### On having more than one way to say the cluster domain

Review feedback on the earlier revision asked for the opposite of this:
fold the `keda:` block into `global.clusterDomain` so the platform has
one answer rather than two added a fortnight apart for the same reason.
The instinct is right and I am not doing it here, because `global` is
the channel that carries the cost. A key under it lands in the values of
nearly every release, one vendored chart already ends its render on it,
and the next chart with the same gate breaks on the day someone adds a
second key. The subchart key has no reach: the chart that declares it
reads it and everything else ignores it, which is how the KEDA domain
already travels.

That does leave the repo with several ways to tell a chart its domain,
and that complaint stands. What it avoids is a shared channel whose
blast radius grows with every new user of it. Whether the platform
should grow one general mechanism here, and which one, is worth settling
on its own rather than inside a bug fix.

### Why the stuck Job is a manual step and not a migration

The release note asks operators to delete a wedged
`<release>-securityconfig-update` Job once. On an existing OpenSearch
installation that Job is already stuck, so this is a step for everyone
the fix is for, and `packages/core/platform/images/migrations/` is the
mechanism built for exactly that. It is not in this PR because the
migration deletes Jobs in tenant namespaces, which is a wider grant and
a wider blast radius than the one key this change adds, and the fix
works without it. If it should ride the same release, it is a separate
commit and I will add it.

### Upgrade behaviour on a running cluster

Read from the operator source at v2.8.0. `DNS_BASE` is read on every
call through `helpers.ClusterDnsBase`
(`pkg/helpers/constants.go:35-41`), so the manager restart is the whole
switch. On a cluster that already runs OpenSearch clusters:

- Certificates are not re-issued. Transport and http certs are generated
only when their Secret is missing (`pkg/reconcilers/tls.go:272-295`,
`490-518`), per-node transport certs only for a pod with no entry in the
Secret (`tls.go:373-377`), the admin cert only when missing, expiring
within five days or not signed by the CA (`tls.go:176-211`). Existing
SANs keep the old suffix. Nothing in-cluster verifies them: the
securityadmin job runs `curl -k` and `securityadmin.sh -nhnv`
(`pkg/reconcilers/securityconfig.go:34,40,46`), the operator client sets
`InsecureSkipVerify` (`opensearch-gateway/services/os_client.go:77`),
Dashboards run with `opensearch.ssl.verificationMode: none`
(`pkg/builders/dashboards.go:180`).
- OpenSearch node pods do not roll, because no StatefulSet field derives
from the DNS base. Dashboards roll, since `OPENSEARCH_HOSTS` changes
(`pkg/builders/dashboards.go:46-47`).
- A securityconfig Job already stuck in its wait loop is not recreated.
The reconciler keeps an existing `<release>-securityconfig-update` Job
while its checksum annotation matches the securityconfig Secret
(`securityconfig.go:140-146`), and the target hostname is not part of
that checksum. Delete that Job once by hand. The next reconcile
recreates it against the new name, and users and roles start applying
from there.

### Known and not addressed here

The upstream `manager.dnsBase` knob stays usable. The platform writes it
through `spec.valuesFrom`, and a per-package override in
`Package.spec.components[].values` lands in `spec.values`, which Flux
merges on top of the references, so a package that needs a different
domain can still say so and win.

The fallback differs from its neighbours and that is deliberate.
`packages/apps/opensearch` falls back to `cozy.local`,
`packages/system/keda` declares `cozy.local` as its package default, and
this package stays on the upstream `cluster.local`. Inside the bundle
all three receive the same value, so nothing diverges in practice, and
`cluster.local` is the right standalone default for a Cozystack
installed over an existing kubeadm or k3s cluster.

A cluster domain that YAML reads as a number does not survive the round
trip. The platform quotes the value into the Secret, so
`networking.clusterDomain: 123` arrives as the string `123`, but the
upstream template renders it as a bare `value: 123` in the container
env, and an `EnvVar` value has to be a string. Quoting it means patching
the vendored template, which is the thing this approach removes, and an
all-numeric cluster domain is not a usable DNS name to begin with.
Before this change the key was never set from the platform, so the path
is new here even though nothing reaches it.

The vendored `opensearch-operator` Deployment template does not
reproduce from `make update` and has not for a while: a fresh pull plus
the patch differs in three places, and one of them is a live defect. The
committed `nindent 10` on `manager.extraEnv` renders invalid YAML, so
that knob cannot be used at all, while upstream's `nindent 8` works.
Nothing in the tree sets `extraEnv`, so it is latent, and the drift
predates this branch. This PR does not touch that patch at all and does
not carry a fix for it.

Nothing in CI renders a package chart with the values the platform
injects, which is why the smee breakage on the earlier approach had to
be found by hand. That is a gap in the harness.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff, which is one template key in
`packages/core/platform/templates/apps.yaml` and two test files.
`packages/core/platform/values.yaml` is untouched and gains no key, so
the hand-written platform-package table on the website is not affected.
No package, app schema, version enum, CRD, release prefix or naming
convention moves, so the Terraform provider is not reached. The
`networking.clusterDomain` key already exists and its default does not
change, so ansible-cozystack, which passes `--cluster-domain=cozy.local`
to k3s and sets only the networking CIDRs in its platform-package
template, needs nothing: the operator now follows the domain that repo
already configures. talm sets the Talos `cluster.network.dnsDomain` from
its own value and does not write the platform value either. Nothing
under `hack/`, no namespace and no package-tree layout moves, so ccp is
not reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(opensearch-operator): the operator's DNS_BASE now follows the platform cluster domain (`networking.clusterDomain`) instead of the upstream `cluster.local` default, so the securityconfig job and OpenSearch Dashboards reach the cluster on installations that use `cozy.local`. On upgrade, an existing `<release>-securityconfig-update` Job that is already stuck waiting on the old hostname is not recreated by the operator: delete it once by hand and the next reconcile recreates it against the new hostname. Existing certificates keep their SANs and keep working, nothing in-cluster verifies them.
```
2026-09-10 15:21:27 +04:00
Andrei Kvapil 774f788016 ci(agents): check commit trailers on every pull request (#4103)
## What this PR does

`docs/agents/contributing.md` makes a few properties of a commit message
blocking at review: the attribution trailer takes the value `LLM` and
nothing else and appears once, no message carries a session trailer or a
link to a transcript, no line is a generation byline naming the tool.
Nothing checked any of it, so the only reader was a human scrolling `git
log`. That reader is expensive and misses things. One open PR carried
eight commits naming a model in the trailer, another four, all found by
hand after the work was done, each one costing a review round and a
rebase.

`hack/check-commit-trailers.sh` decides all of it from the text, and the
`Commit trailers` job in `.github/workflows/pre-commit.yml` runs it over
the commits between the merge base and the head.

Over all 6062 commits on `main` it rejects 1300, every one a true
positive and no prose false-fires: 1289 attribution trailers naming a
model, 24 `Generated with <tool>` bylines, 15 commits carrying two
trailers, two `claude.ai/code/session_...` links.

### Where it lives

Next to the other per-PR text lint in `pre-commit.yml`. The unit-test
job in `pull-requests.yaml` was the other candidate and is wrong for
this: it is gated on the diff touching code, so a docs-only PR skips it,
and a trailer is wrong or right whatever files the commit changed.

The range is `merge-base..head`, recomputed every run, so a rebase or a
force-push is read as it now is. It never reaches the history on `main`,
roughly a quarter of which carries the old form.

### Decisions worth arguing with

It fails rather than warns. A warning in a job nobody watches guards
nothing.

It is not a required check, and that part is deliberate. Branch
protection on `main` requires `pre-commit` and `E2E Tests`, and `Commit
trailers` is a new context, so a PR can still merge with it red. 77 of
the 200 open PRs carry at least one commit this rejects, so requiring it
today turns those red at their next push. Flipping it is a
branch-protection change once the open set has drained, and somebody
else's call.

Fork PRs are checked like any other. The job runs on `pull_request` with
a read-only token and reads commit messages as data.

A merge commit with no trailer of its own passes. Nothing is required of
a commit, only what is written is judged, so a merge commit that does
carry a bad trailer fails like the rest.

Backport branches are exempt. The backport action cherry-picks merged
commits verbatim, so those messages are history and not the author's to
rewrite. The exemption matches the generated name
`backport-<pr>-to-<branch>`, which is the only signal a `pull_request`
event carries, since the label sits on the original PR. So it is a
convention, not something enforced.

The checker is read from the base branch instead of the checked-out PR.
A `pull_request` event takes the workflow definition from the merge of
base and head, so the job shows up on every open PR the moment it lands,
including the ones whose head predates the script, where reading it from
the head is exit 127 and a red check saying nothing about trailers. It
also keeps the checker out of reach of the PR being measured, though the
job around it is not.

Two shapes are left to the reviewer. A model named in `Co-authored-by:`
needs a list of model names to tell it from a person, and such a list
either misses next month's model or fires on somebody's colleague; most
of the 771 such lines on `main` name a model, so it would be deciding a
lot of PRs. A `Generated-by:` trailer has never appeared here at all.
The byline rule matches a capital `G` for a related reason:
case-insensitively, one line in 25 on `main` is an ordinary sentence
that wrapped onto "generated with cozyvalues-gen".

### Known costs

`hack/check-commit-trailers.sh` matches `hack/[^/]+\.sh$` in
`select-e2e.sh`'s `full_suite_pattern`, so this PR and any later one
touching the script escalates to the full e2e suite. That pattern's own
comment scopes it to the e2e harness, which this script is not, and
`inert_config_pattern` already carries `hack/[^/]+\.bats$`. Escalation
is fail-safe so it is cost rather than breakage, and reclassifying it
wants its own review.

The step drops the explicit `git fetch origin "$BASE_REF"` that
`api-review-gate.yaml` keeps before its own `merge-base`. `fetch-depth:
0` already fetches `+refs/heads/*:refs/remotes/origin/*`, and if that
were ever wrong the failure is a loud `merge-base` error under `set -euo
pipefail`, not a silent pass.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The two entries that could fire
are `cozystack/ccp` and `cozystack/community`, both keyed on a change to
the commit convention itself. The convention text is untouched here,
only its enforcement is new, so both copies stay accurate. `ccp`'s
`package-bump` skill already writes `Assisted-by: LLM`, which this check
accepts.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(agents): a pull request now fails when one of its commits carries an attribution trailer whose value is not `LLM`, a second attribution trailer, a session trailer or transcript link, or a `Generated with <tool>` byline. The check reads only the commits the pull request adds; backport branches are exempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Pull requests now include automated checks for commit attribution
trailers, session links, and generation bylines.
  * Backport branches are excluded from this validation.

* **Documentation**
* Added guidance describing the commit-message checks, accepted
attribution formats, exceptions, and how to run the validation locally.

* **Tests**
* Added coverage for valid and invalid commit-message formats, range
handling, merge commits, and exempted content.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 23:02:10 +02:00
Andrei Kvapil d473454529 chore(labels): add triage/needs-proposal for requests that go through cozystack/community (#4042)
## What this PR does

Adds the `triage/needs-proposal` label to `.github/labels.yml` for
feature requests that need a design proposal in
[cozystack/community](https://github.com/cozystack/community) before any
work starts here. Until now that routing rule lived only in chat, so
such requests kept landing in this repository and were re-filed by hand
with nothing on the issue saying why. AGENTS.md now spells out the
handling next to the rest of the triage guidance: the request is
re-filed in cozystack/community, closed here with the label and a link
to the proposal, and work in this repository starts from a new issue
once the proposal is accepted.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

The diff touches `.github/labels.yml` and `AGENTS.md` only; neither file
appears in the downstream trigger map in `docs/agents/contributing.md`.

### Release note

```release-note
chore(labels): add triage/needs-proposal for feature requests that need a design proposal in cozystack/community before work starts here
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Chores**
- Added a new triage label for issues that require a design proposal
before implementation.
- Documented the process for submitting, reviewing, and accepting
feature proposals before work begins.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 22:59:56 +02:00
Andrei Kvapil 3cb9f2179c fix(ci): map the recurring app scopes to their area labels (#4121)
## What this PR does

The auto-labeler maps a Conventional Commits scope to an `area/*` label.
Scopes that PR titles use for apps under `packages/apps` were missing
from that map, so a PR whose title carries only such a scope gets
`area/uncategorized` and the area has to be set by hand afterwards.

Each scope added here clears the bar the map's own comment sets for a
new entry, three or more occurrences over the whole PR history in both
title forms the labeler parses: `vm-instance` 14, `nats` 10, `rabbitmq`
8, `vpc` 8, `foundationdb` 5, `qdrant` 4, `kubernetes-nodes` 3,
`vm-disk` 3. Each also has an unambiguous home among the labels that
already exist. `area/virtualization` names `cdi`, and `vm-disk` renders
a `DataVolume` while `vm-instance` renders a `VirtualMachine`.
`area/kubernetes` is the tenant Kubernetes app and `kubernetes-nodes` is
the worker half split out of it. `area/database` is described as managed
databases and already lists `kafka`, so `rabbitmq`, `nats`, `qdrant` and
`foundationdb` sit there without stretching it. `vpc` renders `Vpc` and
`Subnet`, and `area/networking` already names `kube-ovn`.

Replayed the map before and after this change over the whole PR title
history: 53 titles come out with a different label set. Each of them
gains one `area/*` and loses nothing. 48 stop falling back to
`area/uncategorized`; the other 5 already had an area from a second
scope in the title and now carry both.

A scope that clears the bar but has no unambiguous home is left out
rather than pushed into the nearest label. AGENTS.md asks for a new
`area/*` in that case, which is a separate decision. Those, and the
scopes still below the bar, are collected in #4122 along with the
measurement and two bits of drift found on the way. `opensearch` is not
that case: its home was never in question, it was below the bar when
this PR opened and crossed it while the PR waited. It stays in #4122
rather than being folded in here.

### Screenshots

No UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The diff touches `.github/workflows/pr-labeler.yaml` only. The
nearest triggers are the ccp and community entries on the commit
convention in `docs/agents/contributing.md`, and that file is not
touched here: the scope table there is a summary that already lags the
workflow, and the workflow is named as the full mapping.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(ci): more of the app scopes used in PR titles now map to an `area/*` label instead of falling back to `area/uncategorized`
```
2026-09-09 22:59:51 +02:00
Aleksei Sviridkin b083d8e9e6 test(platform): pin the opensearch-operator DNS base wiring
Two ends of the same path. The platform suite pins that the shared
values Secret carries networking.clusterDomain to the operator under
the subchart key, and the package suite pins that a domain set on that
key reaches the manager container as DNS_BASE, with the upstream
cluster.local default intact when nothing sets it.

Every pattern in the platform suite anchors dnsBase to its
opensearch-operator and manager parents rather than matching the key
anywhere in the payload. An unanchored pattern is satisfied by any
sibling block that happens to carry the same domain, which leaves the
assertion green after the block it is meant to pin has been deleted.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-09 23:57:29 +03:00
Aleksei Sviridkin 857ad4a128 fix(opensearch-operator): resolve in-cluster names on the platform domain
The operator reads DNS_BASE and builds every in-cluster name it needs
from it: the securityconfig job target, its own OpenSearch API client
URL, the Dashboards host, and the SANs it writes into node
certificates. The vendored chart defaults that to cluster.local, so on
a cluster whose domain is cozy.local the securityconfig job never
resolves its target, the .opendistro_security index is never created,
and the OpenSearch nodes never go Ready.

The domain the platform already holds could not reach the operator
before: _cluster stops at each package's own templates and never
reaches a vendored subchart, and the wrapper's values.yaml is not
templated. Carry it in the shared values Secret under the subchart key
instead, the same way the KEDA cluster domain travels, so the vendored
chart picks it up with no patch and a package rendered without the
Secret keeps the upstream cluster.local fallback.

A global key would have worked for the operator but reaches every
release on the cluster, and the vendored smee chart in bootbox cannot
render once global is non-empty. Scoping the value to the subchart
that consumes it avoids putting every other package on that path.

The diagnosis and the first working fix came from fuad00 in an earlier
pull request.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-09 23:56:12 +03:00
Alexey Artamonov 2d1e425339 docs(postgres): correct the KEDA-guard rationale and note the gradual client-side rollout
The KEDA-missing fail-closed guard's comment still described the replaced
drop-instances design ("db.yaml drops spec.instances ... CNPG collapses to 1");
under the shipped seed design db.yaml seeds instances at max(replicas,
effectiveMin), so a skipped ScaledObject leaves a fixed-size cluster with no HPA
- autoscaling silently absent, not a collapse. Correct the rationale. Also note
on the cozyrds apply-strategy annotation that the client-side switch rolls out
per-app as each Application is next written through the aggregated API, not
fleet-wide at upgrade (the ApplicationDefinition reconciler syncs only
chartRef/valuesFrom/labels onto a live HelmRelease). Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-09 21:36:54 +03:00
Alexey Artamonov 154548f57c fix(cozy-lib): reject a negative minReplicaCount and cover the annotations parameter
cozy-lib.keda.scaledObject only checked max >= min, so an all-negative or
mixed-sign pair (min -5, max -3) passed the ordering guard and rendered a
ScaledObject with a nonsensical floor. Reject minReplicaCount < 0 before the
ordering check. Also add a cozy-lib-tests case for the documented annotations
parameter, which had no caller and no coverage. Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-09 21:36:54 +03:00
Alexey Artamonov 1f2ff31470 fix(postgres): pin job=kube-state-metrics on the autoscaler read-load query
The read-load trigger query joined on kube_pod_labels without pinning
job="kube-state-metrics", while the sibling DatabaseAutoscaler* alert rules
already pin it. A second exporter emitting a kube_pod_labels series (or a
duplicate scrape) would multiply the join and distort the desired-count
arithmetic. Pin the job on every kube_pod_labels selector in the query for
parity with the alerts. Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-09 21:36:54 +03:00
scooby87 afc003c464 feat(postgres): database read-replica autoscaling via KEDA (#3954)
## What this PR does

Adds automatic horizontal autoscaling of a PostgreSQL database's **read
replicas** in response to load, implementing the merged design proposal
[Database Horizontal
Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling)
(community#44). The mechanism is entirely stock — no bespoke operator,
no new CRD:

- **KEDA as an optional platform component** (`packages/system/keda`,
wired into `core/platform` as a source + system-bundle entry). It claims
the `external.metrics.k8s.io` APIService that nothing served before.
- **`cozy-lib.keda.scaledObject` helper**
(`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA
`ScaledObject`, fail-closed on missing/invalid arguments.
- **`packages/apps/postgres` autoscaling** — an `autoscaling` values
block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`,
`maxReplicationLagSeconds`, `dryRun`, `transition`) validated by
`values.schema.json`; under active autoscaling the chart renders a
`ScaledObject` and seeds `spec.instances` to a constant floor
(`max(replicas, effectiveMin)`), while the postgres release is forced to
client-side (two-way) Helm apply so the seed is a merge no-op on upgrade
and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster`
scale subresource — is not reverted.
- **Single-value metric (§1)** — one PromQL query returns `Σ(read load
over standbys) + target`, so a stock HPA computes `desired =
ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`,
with no per-pod emission.
- **The four static brakes (§5)** — quorum floor `max(minReplicas,
maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down
pacing (`Pods/1/600s`), a replication-lag clamp folded into the query
(freeze both directions while lag exceeds the threshold and the primary
is writing, with `max_over_time` hysteresis), and dry-run (render
paused).
- **Two-phase enable/disable migration** via a `transition` sub-flag:
phase 1 stages `spec.instances` to the live count and brings KEDA up
with its HPA paused in both directions; flipping to phase 2 hands the
runtime count to KEDA, and because the seed re-renders the same count
the flip is a merge no-op with no dip.
- **Observability** — a database-autoscaling Grafana dashboard and
PrometheusRule alerts.

**Live validation (dev cluster, CNPG 1.30.0).** The full path works
end-to-end: the CNPG `Cluster` scale subresource serves
`status.selector`, KEDA renders the HPA, the metric reaches it
(`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2
= 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query
returns the base value on live VictoriaMetrics. Two packaging fixes were
needed and are included: KEDA must run on the cozystack `cozy.local`
cluster domain (the upstream `cluster.local` default makes the metrics
gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`),
and the operator's client-go throttle is raised so a single-replica
operator does not lose its leader lease under scale-subresource polling.

Implements #3953.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two repositories are affected; both are follow-ups to open once
this lands (and after its dependency #3951 merges and the values surface
is final), left unticked here for a maintainer to confirm rather than
claimed with a speculative link:

- **cozystack/terraform-provider-cozystack** — this adds the nested
`autoscaling` block to `packages/apps/postgres/values.schema.json`. The
provider hand-models each app's fields (schema + model +
expand/flatten), so it needs the new block to expose it in Terraform.
- **cozystack/website** — KEDA is a new platform component
(`guides/platform-stack`), and the postgres reference page gains the
`autoscaling` params. The managed-app reference is regenerated from
`README.md` by the release bot; the platform-stack component list is
hand-maintained.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: to open (new platform component + postgres autoscaling
params)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: to open (new `autoscaling` block in postgres
values.schema.json)

### Depends on

- #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG
`Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves
`status.selector`; without it the HPA fails with `InvalidSelector`. This
PR should not merge before #3951.

### Release note

```release-note
feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added configurable PostgreSQL read-replica autoscaling with KEDA.
* Added CPU and read-connection metrics, replica bounds, replication-lag
safeguards, dry-run mode, and transition handling.
* Added optional KEDA platform support, Grafana autoscaling dashboard,
and Prometheus alerts.

* **Documentation**
* Clarified integer thresholds, prerequisites, configuration, and
transition guidance.

* **Bug Fixes**
* Improved validation and actionable errors for unavailable KEDA or
invalid settings.

* **Tests**
* Expanded coverage for autoscaling, observability, package selection,
and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 21:28:34 +03:00
Andrey Kolkov 92edfb810b fix(backups): run the strategy bats under bash; move template essays to {{/* */}}
The behavioural bats test ran the rendered `set -euo pipefail` script via
`sh`; the CI unit-tests runner's /bin/sh is dash, which rejects `-o pipefail`
at line 1 and reddened the whole suite (production is busybox ash, which
supports it, so the script itself is fine). Run it via `bash`, matching ash
semantics and the hack/migration-50-etcd-adopt.bats precedent.

Convert the multi-paragraph `#` rationale blocks in the shipped strategy
template to `{{/* */}}` so Helm strips them at render instead of copying the
prose verbatim into every rendered Strategy CR, where it drifts as the code
changes. Short one- to three-line guard comments stay as `#`.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 19:23:37 +04:00
Andrey Kolkov 9eb7eeaca5 test(backups): behaviourally pin the shipped Redis strategy script
helm-unittest renders this template but asserts only its structural
decisions, so blanking the object key, breaking the rfs-redis- sentinel
prefix, or disabling the cleanup guard each left the suite green. Add a
bats test (auto-discovered by make bats-unit-tests) that extracts the
rendered script and EXECUTES it with stubbed redis-cli / curl: cleanup
must DELETE its object and never reach the restore FLUSHALL against the
live master, backup must resolve rfs-redis-<app> and upload under the
recorded object key, and an unrecognised mode must fail closed. Each of
those three mutations now turns the suite red.

Also correct the endpoint doc row: the Redis Job always connects over
https (the projector strips the scheme and the script prepends it), so
its TLS is not scheme-driven the way MariaDB/FoundationDB is.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 18:34:37 +04:00
Andrey Kolkov 1289f522fa fix(backups): unbreak install, kill destructive cleanup, reach the resource knob
Store the Redis strategy Template schemaless, as the Rabbitmq strategy
already does: without the markers controller-gen emitted the full
PodTemplateSpec schema and the CRD reached ~602 KB, pushing the chart's
Helm-release Secret past the 1 MiB cap so backupstrategy-controller failed
to install at all.

Give the example strategy a cleanup branch, and both scripts an explicit
`elif restore` / `else exit 1`, so a deleted Backup rendered with
Mode=cleanup deletes its object instead of falling into the restore path
and FLUSHALLing the live master.

Forward a backupStrategyController block through the platform bundle
alongside backupStorage, so redisBackupResources — the restore-Pod sizing
knob the loader-not-ready message points at — is reachable via
spec.components.platform.values.backupStrategyController, not merely
settable on a Helm-managed Strategy CR that reverts on the next upgrade.

Correct the endpointCASecretName doc row to match the now-optional mount,
and `int` the activeDeadlineSeconds interpolation so a large values-file
override renders as an integer, not exponent form.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:53:50 +04:00
Andrey Kolkov 651a6e3023 fix(backups): harden the redis restore AOF wait and image build
Wait for the throwaway loader's AOF rewrite to finish - no rewrite in
flight, last one ok, base AOF on disk - instead of a bare
aof_rewrite_in_progress:0 check, which right after `appendonly yes` can
fall through before the scheduled rewrite starts and replay an absent or
half-written base AOF. Same fix in the example strategy.

Declare the package image targets PHONY and serialize image-redis-backup
after image-backupstrategy-controller so their `yq -i values.yaml` digest
stamps cannot race under `make -j`. Run the demo cleanup under `set -e`
(every delete is already --ignore-not-found) so a real API failure aborts
instead of printing "Cleanup complete" over a half-torn-down demo.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 16:23:12 +04:00
Andrey Kolkov b4d621f4f7 fix(backups): delete Redis backup objects on cleanup and size the restore Pod
Delete the backup object when its Backup is removed. The Redis driver
owns its S3 object (one per BackupJob, no engine retention prunes it), so
it now spawns a cleanup-mode Job that deletes the object and holds the
Backup's finalizer until it completes, mirroring the Rabbitmq driver.
Without this a retention-pruned Plan grew the bucket unboundedly and the
key was lost with the CR.

Size the strategy Pod from backupStrategyController.redisBackupResources
so an admin can raise the restore ceiling (the throwaway loader holds the
whole dataset in memory) via chart values, which a direct edit of the
cluster-scoped, Helm-managed Strategy CR could not survive. The
loader-not-ready diagnostic now names the memory limit, the likeliest
cause, instead of only the image version.

Make the endpoint CA volume optional: nothing projects that Secret into
the app namespace, so a required mount wedged every backup Pod on
FailedMount; a missing ca.crt now falls through to the system CA store
and fails the handshake fast, which is not the same as skipping verify.

Promote the Pod-level activeDeadlineSeconds to the Job so a wedged run is
bounded once, not (backoffLimit+1) times. Use find, not ls, for the base
AOF so its no-match diagnostic is not dead code under pipefail.

Pin the object key reaching the rendered Job: both fixtures now render
--key={{ .ObjectKey }} and assert it, so zeroing the key (which left the
package green) is caught. Document .ObjectKey and the cleanup mode in the
CRD render-context, list cozy-default-redis among the Helm-managed CRs,
and mirror the CA/TLS and find fixes into the example strategy.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-09 12:19:02 +04:00
Andrey Kolkov 56e10b1580 fix(backupstrategy-controller): gate the default MongoDB Strategy on its CRD (#3980)
## What breaks today

Upgrading any existing cluster to a build that carries the new MongoDB
backup driver wedges the `backupstrategy-controller` HelmRelease
permanently, with `resource mapping not found for name:
"cozy-default-mongodb" ... no matches for kind "MongoDB" in version
"strategy.backups.cozystack.io/v1alpha1"; ensure CRDs are installed
first`.

This chart ships its `strategy.backups.cozystack.io` CRDs as ordinary
templates, and Helm resolves every document in a release manifest
through the cluster's RESTMapper in `Build()` **before** it applies any
of them. A Strategy CR whose CRD is not served yet therefore fails
`Build()`, so the release applies **zero** objects — including the CRD
that would have made the mapping resolve — and every retry renders the
same manifest. Nothing recovers on its own.

## Why the gate that exists does not cover it

The chart already knows this trap: it is why every default Strategy
gates on the bucket name. That gate covers the **first install** of a
fresh cluster, where nothing is resolvable yet and no Strategy renders
at all. It cannot cover shipping a **new kind** to a chart that is
already installed — there the bucket name resolved releases ago, so the
first upgraded revision renders the new CRD and the new Strategy CR
together and wedges.

## The fix

Gate the CR on its CRD actually being served, through a `crdEstablished`
helper. The helper asks about the **CRD object** rather than listing the
custom kind, because `lookup` on a kind the apiserver does not serve is
a render *error*, not an empty result, which would trade this wedge for
a different permanent failure; `apiextensions.k8s.io/v1` is always
served, so that lookup can only answer yes or no. It requires
`Established=True` rather than mere existence, because that is when the
kind enters the RESTMapper.

Convergence afterwards is the mechanism the bucket gate already relies
on, and is **not** the HelmRelease interval: once the CRD is served, the
controller's `DefaultObjectsGate` sees the kind mapped and the CR absent
and forces a real Helm upgrade whose render includes it. Kinds that do
not map at all are skipped there (`meta.IsNoMatchError`), which is what
stops the gate forcing upgrades against a render that cannot yet produce
the object.

`bucketNameOverride` short-circuits the CRD lookup for the same reason
it short-circuits the bucket lookup: it marks an offline render — `helm
template`, a CI diff, a unit test — where there is no apiserver to ask.
Live deploys go through Flux and never set it.

## How it was found

The release upgrade E2E lane in #3276, on the `v1.6.2 → main` path.
`mongodbs.strategy.backups.cozystack.io` is the only definition added to
this chart since the tag, so this is reachable only by upgrading an
existing install — no install-only suite can see it.

## Verification

Reproduced and fixed against a live cluster that happens to be in
exactly the pre-upgrade state (the seven older strategy CRDs present,
`mongodbs` absent). `helm template --dry-run=server` of main's chart
fails with the error above and produces zero documents; the same render
with this change succeeds with 21 documents and the Strategy correctly
withheld. The helper's two branches were probed the same way against
that cluster: a served CRD answers `true`, an absent one answers empty.

The new unit case pins the gate offline through the external-S3 path,
where the bucket name resolves without the offline override so that only
the CRD gate can hold the CR back. Removing the gate turns exactly that
case red; the suite is 19/19 green with it.

```release-note
Fixed a permanent failure of the backupstrategy-controller release when upgrading a cluster to a version that adds a new backup-strategy kind: the default MongoDB Strategy is now withheld until its CRD is served, instead of failing the whole release with "no matches for kind".
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved backup strategy rendering during upgrades by waiting until
the required MongoDB resource definition is available.
* Preserved offline rendering when an explicit bucket name is provided.
* Prevented invalid default strategies from being generated when the
required resource definition is unavailable.

* **Tests**
* Added coverage for offline rendering and unavailable
resource-definition scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-09 11:17:28 +04:00
Aleksei Sviridkin 9e577a5771 docs(rabbitmq): say that deleting the application leaves the TLS Secrets behind (#4143)
## What this PR does

`packages/apps/rabbitmq/README.md` documented the key material that
survives turning TLS off, but not the same outcome on deleting the
application. The chart renders `Certificate` and `Issuer` objects and
never the Secrets, and cert-manager runs with
`enableCertificateOwnerRef: false`, so a Helm uninstall leaves
`rabbitmq-<name>-ca` (the CA private key) and `rabbitmq-<name>-tls` in
the tenant namespace with nothing referencing them. One paragraph names
the two Secrets and the cleanup command. Documentation only.

### Screenshots

Not a UI change.

### Downstream repositories

A README paragraph in one chart; the generated reference page is built
from `values.yaml` and does not move.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Documentation**
  - Added uninstall guidance for TLS-enabled RabbitMQ applications.
- Clarified that generated certificate Secrets must be manually removed
from the tenant namespace after application deletion.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 19:45:21 +04:00
Aleksei Sviridkin 2055341ba9 fix(opensearch-operator): raise vm.max_map_count on every node from a DaemonSet (#4152)
## What this PR does

OpenSearch will not start below `vm.max_map_count=262144` when the store
may use mmap, and nothing in the platform sets it. The Talos profiles
declare no sysctls and no documented node-preparation path sets one, so
nodes sit at the kernel default of 65530. The app chart asks the
operator to cover that with `spec.general.setVMMaxMapCount`, which
appends a privileged init container to every OpenSearch pod. Tenant
namespaces carry no PodSecurity label of their own, so they take the
cluster default, which Talos ships as baseline through the apiserver's
`PodSecurityConfiguration`. That default refuses the container and the
StatefulSet never produces a pod. On vanilla kubeadm, kind or k3s the
default is `privileged` and the same release comes up, which is why this
only bites on a Talos install.

The value cannot come from the pod at all. The kernel does not namespace
`vm.*`, so kubelet rejects it in `securityContext.sysctls`, and refuses
to start if it appears in `--allowed-unsafe-sysctls`. That leaves node
configuration or a privileged pod, and a privileged pod belongs in a
system namespace rather than in every tenant.

This adds a DaemonSet to the `opensearch-operator` package that sets the
value once per node from a privileged init container, and marks the
component privileged so the platform labels `cozy-opensearch-operator`
with `enforce=privileged`. The init container writes to `/proc/sys` and
nothing else, with no hostPath. The pod restarts with the node and
reapplies the value, so there is nothing to persist.

It raises the limit to a floor rather than assigning it, which matters
more than it looks. `vm.max_map_count` is node-global, and a stock
Ubuntu 24.04 node already reads 1048576 from a `sysctl.d` drop-in. An
unconditional write would drop that to 262144 for every workload on the
node, on every boot, and undo whatever the operator set it for. So the
init container reads the current value first and writes only when it is
below the threshold.

The idle main container runs its sleep under a shell that traps SIGTERM,
so a pod exits on its own when the DaemonSet rolls or a node drains,
instead of ignoring the signal and being killed after the full grace
period.

This does not fix OpenSearch on its own. The app chart still requests
the operator's init container through `setVMMaxMapCount: true`, so a
release stays blocked until the flip to `false` lands with #2682, after
this merges. The DaemonSet has to be in place first: flipping it without
one would trade the admission failure for a failed bootstrap check.
Carrying the sysctl in the node contract instead, across the Talos
profiles and the downstream node-prep paths, is tracked in #4144; that
is what would let this DaemonSet be retired.

Two things change on upgrade, and neither is visible from the diff. One
small DaemonSet pod appears per node in `cozy-opensearch-operator`, and
that namespace gains `pod-security.kubernetes.io/enforce=privileged`.
That label has no finer granularity than the namespace, so the operator
Deployment sitting there leaves the cluster default too, not just the
DaemonSet. Keeping the manager at baseline would mean a namespace of its
own and another component entry, which did not look worth it, but it is
a real cost rather than an accident. The operator is in the `paas`
bundle's always-on list, so this lands on every node of every paas
installation and raises a node-global kernel tunable there, whether or
not anyone ever creates an OpenSearchCluster. That is the price of a
node-level sysctl having no cheaper carrier, and it is worth an explicit
decision rather than being noticed later.

The second is `cozypkg add`, which gates privileged components behind a
confirmation or `--allow-privileged`. The gate runs over the resolved
dependency closure, and `opensearch-application` dependsOn the operator,
so installing the app that way now asks one more question. This is not a
new break for scripts: `add` already prompts for a variant on every
package it resolves and fails on EOF, so it was never usable
non-interactively in the first place. Platform-bundle installs go
through the reconciler and are unaffected.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. It adds a template, a test and a values key under
`packages/system/opensearch-operator/`, plus one field in
`packages/core/platform/sources/opensearch-operator.yaml`. No new app
package, no app schema or version enum, no `ApplicationDefinition`, no
installer value, no namespace rename, no variant or bundle, no `hack/`
path or make target, and no node prerequisite in
`hack/e2e-prepare-cluster.bats`. Nothing in the map matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(opensearch-operator): set `vm.max_map_count` on every node from a DaemonSet in the operator's namespace, so OpenSearch no longer depends on a privileged init container that PodSecurity baseline refuses in a tenant namespace. Upgrading adds one small DaemonSet pod per node and labels `cozy-opensearch-operator` with `pod-security.kubernetes.io/enforce=privileged`; installing the package with `cozypkg add` now requires confirmation or `--allow-privileged`. Existing OpenSearch releases are unchanged until a follow-up stops the app chart requesting the operator's own init container.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- OpenSearch Operator installation now runs with the required privileged
configuration.
- OpenSearch deployments automatically ensure nodes meet the required
`vm.max_map_count` setting, improving startup reliability.
  - Existing higher kernel values are preserved.
- Added support for configuring the sysctl image, including use with
air-gapped registries.

- **Tests**
- Added coverage to verify privileged installation, node configuration,
security settings, scheduling behavior, and image customization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-08 19:45:14 +04:00
Andrey Kolkov 84fe2ecf6f docs(backups): mirror the redis TLS fix into the example strategy
The demo strategy CR states its in-Pod script is identical to the shipped
cozy-default-redis strategy, but it kept the plaintext Sentinel and master
dial after the shipped template gained server-side TLS. Add the same
optional redis-<app>.ca-cert mount and RTLS auto-detection so a tenant who
copies this example to back up a tls.enabled Redis does not reproduce the
plaintext-against-a-TLS-only-listener failure.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-08 19:13:50 +04:00
Alexey Artamonov 9511f56307 fix(postgres): require service.namespace.svc shape in the autoscaling SSRF host guard
The serverAddress host regex accepted any dotted tail ending in .svc, so a
single-label host like foo.svc passed the in-cluster-DNS check. It then reached
the namespace-pin, whose extraction regex left the unresolvable host unchanged
and rejected it as a namespace outside the tenant subtree - the right outcome by
the wrong path, with a misleading message. Anchor the host to two DNS labels
before .svc so it is refused at the host check with the actionable message, and
drop the stray dot the old label class allowed.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-08 18:06:34 +03:00
Andrey Kolkov 45a2f54c62 feat(backups): back up TLS-enabled Redis apps via the default strategy
The shipped cozy-default-redis strategy connected to Sentinel and the
master in plaintext, so a BackupJob against a Redis app with tls.enabled
(the operator moves both to a TLS-only listener) failed at master
resolution. Mount the app CA the operator publishes at
redis-<app>.ca-cert as an optional Secret, and dial redis-cli with
--tls --cacert exactly when its ca.crt is present, so a non-TLS app
keeps the plaintext path and the throwaway restore loader stays
local-only. Covers the default authClients: no; mutual TLS is out of
scope, since the platform issues no client certificate.

Assisted-By: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-08 18:47:13 +04:00
Andrey Kolkov 2d82c1ea26 test(backups): keep main's redis-tls e2e Tests; fast-fail redis backup waits
The branch was cut before main gained the redis-tls and
redis-tls-flip-rejected Chainsaw Tests, so its copy of the redis
chainsaw-test.yaml carried only the plaintext Test plus the new
redis-2-backup-roundtrip. Carry main's two TLS Tests here too, so a conflict
resolution at merge time cannot drop the TLS end-to-end coverage regardless
of which side it takes. The file still needs a rebase onto current main to
merge cleanly; this only removes the risk of losing the Tests.

Also pass the terminal Failed value to the BackupJob/RestoreJob phase waits
so a failed run aborts immediately with a legible cause instead of polling to
the 600s timeout.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-by: LLM
2026-09-08 17:57:06 +04:00