The read-load query's base term is floored with `or vector(0)`, but the
replication-lag brake's frozen additive term (count(kube_pod_labels) * target)
was not. PromQL `+` is a set intersection, so if count(kube_pod_labels) is
momentarily empty — a kube-state-metrics scrape gap or KSM rollout — the frozen
term is empty and the whole `base + frozen*braking` sum collapses to empty,
defeating the base term's own graceful degradation. KEDA's prometheus scaler
(ignoreNullValues: true by default) then reads that as 0 and pins desired to
minReplicaCount, ignoring real read load exactly when metrics are flaky. This
hits the default config (maxReplicationLagSeconds: 30, brake active). Floor the
frozen summand with `or vector(0)` so base + 0 = base, and assert the floor in
the lag-brake unittest.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Cover the load-bearing replica selector that both read-load queries depend on:
the join must restrict to standby pods via label_cnpg_io_instance_role="replica",
the exact key kube-state-metrics exposes for CNPG's cnpg.io/instance_role. A
wrong key (e.g. a camelCase label_cnpg_io_instanceRole) would match no series,
floor the read term to 0, and silently pin desired to the minimum - the suite
previously never asserted the selector at all. Verified live on dev9 (CNPG
1.30.0) that the snake_case key is the one present in kube_pod_labels and the
camelCase form does not exist. Adds the assertion for both the ReadConnections
and ReadCPUUtilization paths.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Two review findings:
- The enablement note claimed the two-phase `transition` flow avoids the
count dip when enabling autoscaling on an existing cluster. It does not:
helm-controller upgrades with Helm's three-way merge (not SSA), which keys off
the old-vs-new manifest diff, not KEDA's live /scale value. At phase 2 the
chart stops rendering the staged static count, so the field is deleted (no
quorum -> CNPG defaults to 1, a brief collapse) or changed (quorum -> rebased
to effectiveMin), then the warm KEDA restores it. Rewrite the docs to state the
flow reduces (KEDA is pre-warmed, no cold-start) but does not eliminate the dip,
and spell out the suspend + scale-subresource pin that actually makes the
handoff dip-free. This is the SSA-vs-three-way-merge handoff the design proposal
left for the PoC to settle.
- Pin metricType: AverageValue on the Prometheus trigger in
cozy-lib.keda.scaledObject. The desired = ceil(query / threshold) identity
holds only for an AverageValue external metric; it was relying on KEDA's
default. Make it explicit and assert it so a re-vendor or default change cannot
silently break the arithmetic.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Narrow the previous unconditional seed to a conditional one, keeping design
proposal §3 (chart omits the field, HPA is the sole writer) intact for the
default path. Without synchronous quorum (maxSyncReplicas=0) CNPG's defaulted
instances=1 is admissible, so the field is omitted exactly as §3 specifies and
KEDA owns it via /scale with nothing contested. Only when maxSyncReplicas>=1 -
where an omitted/defaulted instances=1 fails the admission webhook
("maxSyncReplicas must be lower than the number of instances") - does the chart
seed the quorum floor (effectiveMin) so a fresh Cluster is admitted; the constant
re-render keeps three-way merge from reverting KEDA's higher live value.
This confines the deviation to the case §3 did not consider (sync quorum), so no
design-proposal amendment is needed for the documented path. Verified live on a
dev cluster: the quorum render (instances=2, maxSyncReplicas=1) and the
non-quorum render (instances omitted) are both admitted by the CNPG webhook. Adds
a render test for the non-quorum omit path alongside the quorum seed tests.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Under active autoscaling the chart previously omitted .spec.instances entirely,
letting the KEDA HPA own it via /scale. But CNPG defaults an absent
.spec.instances to 1, and its admission webhook rejects a Cluster where
maxSyncReplicas >= instances ("maxSyncReplicas must be lower than the number of
instances"). So a fresh database created with autoscaling.enabled: true AND
synchronous quorum (maxSyncReplicas >= 1) was rejected at admission before KEDA
could ever scale it to effectiveMin (= maxSyncReplicas+1) — the whole tenant
HelmRelease failed. The default maxSyncReplicas: 0 path was unaffected, which is
why it went unnoticed. Verified live: a rendered Cluster with maxSyncReplicas: 1
and an omitted instances is rejected by the dev-cluster webhook; with the seeded
floor it is admitted.
Render instances: effectiveMin under active autoscaling instead of omitting it,
so a fresh Cluster starts at a valid, quorum-satisfying count. Because the chart
emits the same constant floor on every reconcile, Helm's three-way merge is a
no-op on the field and never reverts KEDA's higher live value, so the HPA still
owns the runtime count. Update the enablement docs (a new DB can enable directly
even with synchronous replication; existing DBs above the floor rebase to it
unless the two-phase transition flow is used) and add render tests for the fresh
create + maxSyncReplicas>=1 case.
Addresses the fresh-create-with-quorum admission gap found in branch review.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
autoscaling.enabled is a tenant-facing knob, but KEDA is an optional platform
package that is not installed by default. A tenant enabling autoscaling on a
cluster without KEDA would render a ScaledObject against an unknown kind and the
whole database HelmRelease would fail with an opaque `no matches for kind
"ScaledObject"`. Guard the ScaledObject template on
`.Capabilities.APIVersions.Has "keda.sh/v1alpha1"` and stop with an actionable
message instead. Skipping the render is deliberately NOT an option: db.yaml
already drops the static .spec.instances under active autoscaling, so a missing
ScaledObject would let CNPG collapse the cluster to 1 instance. Document the
KEDA prerequisite on the `enabled` field (README/schema).
Also reject a non-positive autoscaling.target at render time: it is a plain
integer now, but the desired-replica formula divides the read load by it, so 0
is undefined and negatives invert scaling. cozyvalues-gen has no minimum
directive, so the bound is a template guard.
Add unittests for both guards (a dedicated suite renders without the KEDA CRD so
the fail-closed path is exercised; the main suite now advertises keda.sh/v1alpha1
via capabilities).
Addresses IvanHunters (KEDA-absent) and CodeRabbit (non-positive target) review
feedback on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Two review findings on the generated API:
- autoscaling.target was a resource.Quantity, which accepts suffixed values
like `150m`. The chart splices target straight into the PromQL query and the
KEDA threshold, where `150m` means 150 minutes to PromQL and is not a usable
KEDA float threshold. Retype it as a dimensionless integer so the schema
rejects suffixed input - helm unittest's schema validation now enforces it,
and the existing autoscaling suite exercises the integer contract.
- ConfigSpec carried a root `metric` field (and spec.metric in the postgres-rd
schema) that nothing reads - the chart consumes only autoscaling.metric - so
setting it silently did nothing. Remove the dead root param, keeping the
ReadMetric enum that autoscaling.metric references.
Regenerate types.go, zz_generated.deepcopy.go, values.schema.json, README.md
and the postgres-rd schema; update the autoscaling tests to the integer target.
Addresses CodeRabbit review feedback on #3954.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Wire autoscaling.maxReplicationLagSeconds into the read-load query: while the
max standby lag has exceeded the threshold within the cooldown window and the
primary is actively writing WAL, the query returns currentInstances*target,
pinning desired to the current count and freezing scaling in both directions
(safer than pinning maxReplicas, which would still allow scale-down under
lag). The lag term uses max_over_time as a hysteresis band; the write gate
(WAL-record rate > 0) keeps an idle primary from tripping it. Metrics
verified live on CNPG 1.30.0: cnpg_pg_replication_lag and
cnpg_collector_wal_records exist and carry namespace/pod (the proposal's
cnpg_pg_stat_replication_sent_diff_bytes does not). Also floor the read load
at zero via `or vector(0)` so an idle cluster returns target rather than an
empty result.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Make enabling autoscaling on an existing database count-preserving, and
add a second read-load signal:
- autoscaling.transition (migration phase 1): keep .spec.instances
rendered as a floor (from replicas, which the operator stages to the
live count) AND pause the ScaledObject, so KEDA stands up without
contending with Flux for the field. Flip transition off (phase 2) to
drop the static field and hand the count to KEDA — suspend the
HelmRelease across that apply, since helm-controller uses Helm's
three-way merge (not SSA) and would otherwise race CNPG's default-to-1.
The procedure is documented on the values field.
- autoscaling.metric (ReadConnections | ReadCPUUtilization): the query
helper emits either the active-connections sum or the postgres-container
CPU sum, both + target.
active = enabled && !dryRun && !transition; the ScaledObject is paused
whenever dryRun or transition is set. Covered by helm unittest (phase-1
holds the staged count + pauses; CPU metric query) and regenerated
schema/README/Go types/postgres-rd.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Implement the KEDA-based read-replica autoscaler for postgres per the
merged design proposal (cozystack/community#44):
- cozy-lib.keda.scaledObject: engine-agnostic helper rendering a KEDA
ScaledObject (scale target, effective bounds, prometheus trigger,
scale-down behavior, optional pause), fail-closed on bad input.
- postgres: an `autoscaling` values block (enabled / minReplicas /
maxReplicas / target / maxReplicationLagSeconds / dryRun). Under active
autoscaling the chart drops the static .spec.instances so Flux and the
KEDA-managed HPA no longer contend, and renders a ScaledObject whose
minReplicaCount folds in the synchronous-quorum floor
(max(minReplicas, maxSyncReplicas+1, 2)); when the floor exceeds
maxReplicas, quorum wins. dry-run keeps the static count and pauses the
ScaledObject. The WorkloadMonitor tracks the quorum floor instead of a
stale static value.
Covered by helm unittest (cozy-lib helper: valid render, pause, three
fail-closed guards; postgres: instances dropped, quorum-floored bounds,
quorum-over-max, dry-run pause). Regenerated values.schema.json, README,
Go types and postgres-rd.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The comment sent operators to a ca.crt key in <release>-credentials.
That Secret is rendered by this chart from .Values.users and carries
password entries only, so there is nothing in it to verify a server
certificate against.
The anchor a tenant can reach is the key-free <release>.tenant-ca that
the CA extraction controller publishes from the sentinel this chart
renders, delivered through the tenantsecrets API.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.
1. Archive-vs-recovery serverName collision. The driver set the restored
cluster's WAL-archive serverName equal to the recovery source
(bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
chart pinned the archiver serverName to the release name. In place those
coincide, so the cluster archived onto the source's own WAL prefix and
barman-cloud-check-wal-archive refused to start ("Expected empty
archive"). Fixed by giving the restored cluster a fresh
bootstrap.newServerName, keyed by the RestoreJob UID (unique per
restore, stable across reconciles); the chart archives there while
externalClusters still recovers from bootstrap.serverName. This is what
the manual CNPG strategies hit (serverName defaults to the cluster
name); the cozy-default flow dodged it (serverName is <ns>-<app>).
2. useSystemBucket guard. The restore populates explicit S3 coordinates
(destinationPath / s3CredentialsSecret / endpointCA) but left
backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
incompatible with useSystemBucket" guard failed the HelmRelease upgrade
and the cluster never re-rendered. This is what the cozy-default
(system-bucket) flow hits on an in-place restore. Fixed by clearing
useSystemBucket in the restore patch, since the app is now on the
explicit-creds path. The driver's app view (postgresapp.Backup) gains
the field, without omitempty so a false survives the merge patch.
Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README told a tenant to fetch <release>.tenant-ca, but the release
name carries the postgres- prefix the application definition applies, so
the object is postgres-<name>.tenant-ca. Following the old text returned
NotFound.
Also drop two comments promising a companion ValidatingAdmissionPolicy
that pins sentinel writes to helm-controller. No such policy ships: the
proposal for it was rejected because authorization by username duplicates
RBAC in a more brittle form. The RBAC boundary is the whole control.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds an engine-agnostic controller that publishes a key-free
`<release>.tenant-ca` Secret (only `ca.crt`) for a managed application,
so a tenant can verify the application's TLS without ever seeing a
private key. The source is declared, not guessed: a chart renders a
namespaced `TenantProjection` sentinel (group `internal.cozystack.io`)
naming the CA Secret to lift; the controller watches the sentinel,
extracts `ca.crt`, and writes the projection owner-referenced to the
sentinel, so garbage collection is native. The projection carries the
`internal.cozystack.io/tenant-ca` label the lineage webhook turns into
tenant visibility. Postgres is the first consumer.
The trust boundary is RBAC: no tenant role grants any verb on
`internal.cozystack.io`, so a tenant cannot forge a sentinel. A
companion `ValidatingAdmissionPolicy` that pins the writer to
helm-controller ships separately as defence in depth.
This is the controller-and-CRD slice, split out from #3299 for focused
review. It is the only piece required for the feature to work; the
render-time helper, the writer policy, the capability-gate, and the
release-prefix validation land as their own PRs.
### Downstream repositories
Walked the trigger map against the diff. This adds an internal CRD and
controller and a sentinel to the Postgres chart; no app is added or
renamed and no `values.schema.json` changes, so the typed provider and
Ansible are unaffected. The one to confirm is the Postgres README edit —
if the website mirrors it, it needs a refresh.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
### Release note
```release-note
feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added tenant-facing CA certificate projections for PostgreSQL
deployments.
* Tenant secrets now provide only the CA certificate, excluding private
keys.
* Added support for configuring and monitoring tenant CA projections.
* **Documentation**
* Updated PostgreSQL TLS instructions to retrieve certificates through
tenant secrets.
* **Bug Fixes**
* Improved protection against invalid certificates, secret collisions,
and accidental exposure of private key material.
* Added validation to ensure projected certificates remain synchronized
with their sources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
helm-unittest coverage for the S3 request-checksum fix (was mutation-negative
before): assert spec.instanceSidecarConfiguration.env pins
AWS_REQUEST_CHECKSUM_CALCULATION=when_required on all rendered barman-cloud
ObjectStores — postgres backup + bootstrap-recovery
(packages/apps/postgres/tests/backup_storage_test.yaml) and the Keycloak DB
(new packages/system/keycloak/tests/db_backup_test.yaml). Verified by mutation:
reverting the template change now fails these assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Address review on the S3 request-checksum fix:
- Extend the fix to the platform-managed useSystemBucket=true flow, whose
ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
barmanSidecarConfiguration(). The platform's own default system bucket is
SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.
- Factor the duplicated instanceSidecarConfiguration block (3x across two
charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
The barman-cloud plugin's sidecar uploads via boto3. Since botocore ~1.36
(early 2025) the default RequestChecksumCalculation is when_supported, which
attaches a flexible checksum to every PutObject. Non-AWS S3-compatible
backends (Ceph RADOS Gateway, some MinIO / Cloudflare R2 builds) reject the
accompanying header with "InvalidArgument: x-amz-content-sha256 must be
UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256
value", so every backup/WAL-archive upload fails against them.
Pin the sidecar to when_required via the ObjectStore's
spec.instanceSidecarConfiguration.env (botocore honors the
AWS_REQUEST_CHECKSUM_CALCULATION env var) on both the keycloak system DB and
the postgres app (backup + recovery ObjectStores). when_required is also
accepted by AWS S3 on a plain PutObject, so it is a safe default everywhere.
This mirrors the etcd-operator fix (cozystack/etcd-operator#342, released in
v0.5.3), which applied the same WhenRequired policy to its aws-sdk-go-v2 S3
client for the same Ceph RGW backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
CloudNativePG stores its self-signed CA in <release>-ca alongside the
private key and offers no way to label it, so a tenant cannot be handed
the trust anchor without also being handed the key. Render a
TenantProjection that lifts ca.crt alone into the key-free
<release>.tenant-ca Secret, and select that label from the postgres
tenant-resource definition so the projection is served through the
tenantsecrets API. Document the new retrieval path in the chart README.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The db.yaml plugins-block comment still carried the over-generalized
claim that the pinned postgresql image variants no longer ship the
barman-cloud-* binaries; mirror the scoped wording from
docs/operations/backup-classes.md (only the standard variant keycloak-db
pins lacks them; this chart's bare system-flavor tags still ship them
but the flavor is deprecated upstream).
After the rebase onto main this PR no longer touches linstor-scheduler:
main re-vendored it to upstream chart 0.3.1 / extender v0.3.6, which
keeps the admission webhook and stops stripping unknown pod fields
(including initContainers[].restartPolicy the barman sidecar needs). The
upgrade note claiming the webhook is removed and locality scheduling
becomes advisory is rewritten as a hold-back warning.
Also: afterwards -> afterward (American English, review nit).
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
make -C packages/apps/postgres generate re-pads the Bootstrap parameters
table for the new bootstrap.serverName row; the committed README still
carried the old padding and kept the pre-commit generate check red.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300, non-blocking): the barman-cloud ObjectStore CRD
validates spec.retentionPolicy against ^[1-9][0-9]*[dwm]$, so an operator who
explicitly sets backup.retentionPolicy="" would render retentionPolicy: "" and
the API server would reject the ObjectStore, breaking backup provisioning. Guard
the field in the apps/postgres and keycloak chart ObjectStores with
{{- if .Values.backup.retentionPolicy }} (the Go driver path is already safe via
omitempty). Add a helm-unittest asserting an empty retentionPolicy is omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300): the apps/postgres user-facing docs still
described the removed native spec.backup.barmanObjectStore / archive_command
model. Update the values.yaml @field/@typedef/@param annotations (useSystemBucket,
schedule, S3CredentialsSecret, EndpointCA, endpointCA, the @param backup note,
bootstrap.serverName) and the hand-written README Operations prose to the
barman-cloud plugin + ObjectStore model, then regenerate values.schema.json, the
README parameters table, the api types, and the postgres-rd ApplicationDefinition
schema (which feed the dashboard form help + cozyrd schema). Field names,
defaults, and the storageClass immutability marker are unchanged; only backup/
bootstrap description text changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-ups for the CNPG barman-cloud plugin migration (cozystack#3300):
- docs/operations/backup-classes.md: the 'Upgrade notes from chart-managed
backups' section still described the removed native spec.backup.barmanObjectStore
behavior. Rewrite it for the plugin flow (chart renders spec.plugins + a
barmancloud.cnpg.io/ObjectStore; the driver SSA-applies an ObjectStore and
patches spec.plugins), and document the real upgrade hazard: the
plugin-barman-cloud operator + cert-manager must be present in
cozy-postgres-operator or the Cluster references an unregistered plugin and
backups/WAL archiving stall.
- apps/postgres backup_storage_test.yaml: add a positive bootstrap-recovery
rendering test. Asserts externalClusters[].plugin references
barman-cloud.cloudnative-pg.io with barmanObjectName=<release>-recovery and
serverName on the plugin parameter, and that the -recovery ObjectStore carries
spec.configuration but NOT serverName (the plugin forbids it there). Previously
the recovery path had only fail-fast guard tests, no positive coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
_lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
their post-branch assertion drift is re-ported into the Chainsaw
suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
the Chainsaw suite phrasing
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address review feedback on #3069:
- ttlSecondsAfterFinished 120 -> 3600 so a failed init Job leaves a 1h
debugging window before the TTL controller collects it (the TTL also
reaps failed Jobs, not just successful ones).
- Load templates/init-script.yaml in the unittest suite: the Job's
checksum/config annotation includes that template, so rendering
init-job.yaml in isolation failed in CI. Select the Job via
documentSelector and assert the 3600 TTL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
Drop the explanatory comment added alongside ttlSecondsAfterFinished;
the rationale lives in the PR/commit history.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The post-install/post-upgrade init Job is a one-shot reconciler of
users/databases/roles. Its hook-delete-policy was `before-hook-creation`
only, which never fires on uninstall, so the completed Job and its Pod
lingered in the tenant namespace forever after the Postgres CR was
deleted (#3055).
Add `ttlSecondsAfterFinished: 120` so the Kubernetes TTL controller GCs
the Job and its Pod ~2 minutes after completion, regardless of the Helm
hook lifecycle. This deliberately avoids touching the hook-timeout
semantics that the bootstrap.recovery path depends on.
Add a helm-unittest suite locking in the TTL and the bootstrap-skip
behaviour to prevent regression.
Fixes#3055
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
Rewrite docs/agents/e2e-testing.md for the Chainsaw suite: declarative
asserts replace the until-get/wait backstops, Chainsaw cleanup replaces
test-level EXIT traps, catch blocks replace the `|| { ...; false; }` idiom,
and the TIA/diagnostics sections describe suites + `make test-chainsaw
CHAINSAW_SUITES`. Move the BATS->Chainsaw bullet out of 'in-flight' since it
is now the standard.
Fix references the migration left stale: hack/e2e-apps/ -> hack/e2e-chainsaw/
in the gemini styleguide and the nats/etcd backup READMEs. The
backup-postgres.bats / backup-foundationdb.bats files cited by the postgres
examples, the postgres README, and the foundationdb controller comment never
existed -- drop the false 'CI e2e drives this' claims and describe those
backup flows as the manual reference flows they are.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add the storageclass dynamic-dropdown x-cozystack-options annotation to the
storageClass field of the data apps (openbao, clickhouse, mariadb, redis,
harbor, opensearch, rabbitmq, mongodb, postgres, nats, kafka, qdrant,
foundationdb, http-cache) at the values.yaml source, so the extension is
emitted by cozyvalues-gen instead of hand-edited into values.schema.json.
kafka carries two (Kafka and ZooKeeper structs); foundationdb is nested
under the Storage struct.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Mark fields that reference cluster resources with the x-cozystack-options
vendor keyword so the dashboard renders them as dynamic dropdowns:
GPU, instanceType, instanceProfile, network (vm-instance), GPU/
instanceType (kubernetes nodeGroups), image (vm-disk), storagePool
(bucket), storageClass (all apps) and disks[].name (vm-instance). Drop
the stale static instanceProfile enum.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Resolve the conflict in the generated packages/system/postgres-rd/cozyrds/postgres.yaml by regenerating openAPISchema and keysOrder from the merged values. The CRD now carries both the chart's tls.enabled field and main's backup useSystemBucket / BackupClass-driven fields.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
version is a general deployment parameter, not a TLS setting; relocate
its @param/@enum block back into Common parameters so the generated
parameter reference groups it correctly instead of under TLS.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The generated JSON schema types tls.enabled as a scalar boolean, which
rejects an explicit null value; describe the tri-state unset case as
"omitted" so the wording matches what the schema actually accepts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Remove the jq post-processing patch that widened tls.enabled type to
["boolean","null"] in values.schema.json and the openAPISchema embedded
in the cozyrds CRD. Plain "type":"boolean" is emitted directly by
cozyvalues-gen and is the correct form.
The cozystack-api server unmarshals openAPISchema into
apiextv1.JSONSchemaProps whose Type field is a single string; an array
value silently zeros the field and disables server-side defaulting for
the entire app.
Tri-state semantics are preserved on the Helm template side via
kindIs "invalid": omitting tls.enabled triggers the external-based
fallback; null is no longer a valid input and is rejected by the schema.
Remove two helm-unittest cases that injected tls.enabled: null (now
intentionally rejected) and update the suite comment to match.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The dashboard RBAC intentionally does not grant access to the operator-managed
<release>-ca Secret (it contains ca.key, which would leak the private key).
Advertising it in cozyrds secrets.include created a UX gap: the dashboard
showed a Secret the tenant could never read.
CNPG bundles ca.crt in every user-credentials Secret it creates, so tenants
already have the trust anchor they need via the <release>-credentials Secret
which is covered by RBAC. Update README and the db.yaml comment to point there.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Add a precondition that fails early if cozyvalues-gen no longer emits
tls.enabled.type as the bare "boolean" string the patch step expects.
Without the guard, a generator output that drops or moves the field
would silently create it via the assignment, and the post-patch
equality check would still pass against the synthetic value.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
The README still described the chart-side cert-manager chain that was
replaced when the chart switched to CNPG operator-managed TLS via
spec.certificates.serverAltDNSNames. Rewrite the section to explain the
actual cert ownership boundary (CNPG owns the chain, chart only injects
the external hostname), the trust anchor location (operator-managed
<release>-ca Secret, key ca.crt), and the chart-side semantics of the
tri-state tls.enabled flag. The values.yaml @field comment is updated
in the same direction, with regenerated schema/types.go/cozyrds.
Clarifies that tls.enabled=false only skips the chart-side SAN
injection - CNPG retains its built-in TLS on the wire regardless of
this flag. To force PostgreSQL TLS off entirely, the caller would set
postgresql.parameters.ssl="off" at the CNPG layer.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Drop the templates/certmanager.yaml suite entirely (the file no longer
exists) and rewrite the spec.certificates assertions to match the new
operator-managed flow: certificates block emitted only when both TLS is
on AND external is true, with serverAltDNSNames carrying just the
external hostname.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
CNPG admission webhook rejects spec.certificates that combines a
serverTLSSecret with serverAltDNSNames, and the operator's internal
client-CA generation step expects ca.key on the secret bound to
serverCASecret — a field cert-manager does not produce, since it stores
the private key as tls.key. The previous chart configuration triggered
exactly this path and left the Cluster stuck on
"generating client CA certificate: missing ca.key secret data".
Drop the chart-rendered cert-manager Issuer/Certificate chain and the
spec.certificates.serverCASecret/serverTLSSecret references. CNPG now
auto-generates the self-signed CA and signs server, replication, and
client certs from it, producing the expected per-purpose secrets
(<release>-ca, <release>-server, <release>-replication). The operator's
default SAN list already covers the three built-in services
(rw, r, ro) in four DNS forms each.
The chart still needs to inject the external LoadBalancer hostname when
external=true, since that name lives outside the operator's auto-derived
set. Emit spec.certificates.serverAltDNSNames with that single entry
under that condition only.
End-to-end test on dev13: Cluster reaches "healthy state", probe pod
with PGSSLMODE=verify-full against the operator-managed CA succeeds and
SHOW ssl reports on.
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Add _tls.tpl helper with tri-state resolution logic (nil/false/true)
- Add certmanager.yaml to provision CA and leaf Certificate resources
- Wire TLS listener configuration into db.yaml when TLS is enabled
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Extend PostgresSpec with nullable Tls.Enabled (*bool) for tri-state control
- Regenerate deepcopy and update go.mod with k8s.io/utils as direct dep
- Add TLS fields to values.yaml, values.schema.json (with nullable support)
- Update README and Makefile; propagate TLS field to cozyrds postgres.yaml
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.
The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.
The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.
kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.
A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).
Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.
Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ivan's original schema edits were correct in intent but lived in the
generated artifacts (values.schema.json and the embedded openAPISchema
in postgres-rd/cozyrds/postgres.yaml). `make generate` reverts them
because values.yaml's `@field {map[string]string}` annotation is the
source of truth -- same regeneration trap that caught #2466 (issue #2597).
cozyvalues-gen v1.4.0 (cozystack/cozyvalues-gen#25, merged 2026-05-21)
adds an `intOrString` type alias for map values that emits
`additionalProperties.anyOf: [integer, string]` plus
`x-kubernetes-int-or-string: true`. Move the change to source by
flipping the annotation, then regenerate:
- packages/apps/postgres/values.yaml: `@field {map[string]intOrString}`
with updated description noting integer values are accepted.
- packages/apps/postgres/values.schema.json: regenerated -- now carries
anyOf + x-kubernetes-int-or-string: true.
- packages/system/postgres-rd/cozyrds/postgres.yaml: embedded
openAPISchema regenerated to match.
- packages/apps/postgres/README.md: regenerated type column.
- api/apps/v1alpha1/postgresql/types.go: Parameters is now
`map[string]intstr.IntOrString`.
The result survives future regenerations. The template's
`{{ \$value | toString | quote }}` already coerces both inputs to
quoted strings, verified end-to-end with helm template against both
`max_connections: 100` (int) and `max_connections: "100"` (string) --
identical render output. Dangerous-param block list still rejects
archive_cleanup_command and recovery_end_command additions from the
prior commit.
Co-Authored-By: IvanHunters <xorokhotnikov@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add archive_cleanup_command and recovery_end_command to the dangerous
parameters blocklist in db.yaml. Both are PostgreSQL shell-exec hooks
identical in threat model to archive_command/restore_command and must
not be settable by tenants.
Also update the BLOCKED list in values.schema.json, values.yaml, and
the embedded openAPISchema in cozyrds/postgres.yaml to reflect the
two new blocked parameters.
Additionally update the embedded openAPISchema in cozyrds/postgres.yaml
to allow integer values for postgresql.parameters (anyOf string|integer),
matching the fix already applied to values.schema.json.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The template already coerces all parameter values to strings via
'toString | quote', but the JSON schema had 'additionalProperties: {type: string}'
which caused Helm schema validation to fail before the template ran
when users passed integer values (e.g. max_connections: 100).
Accept both string and integer types in additionalProperties so that
existing configurations with unquoted numbers continue to work.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>