talos.imageFactoryURL is documented for self-hosted factories and mirrors
in air-gapped, rate-limited or flaky-egress environments. Such an endpoint
is normally served with a certificate from an internal CA, and CDI — which
trusts only its system store — refuses the import with "x509: certificate
signed by unknown authority". The DataVolume never completes, no worker
joins, and the release sits in install until it times out.
DataVolume.spec.source.http.certConfigMap exists for exactly this, but the
chart rendered source.http.url alone and no value reached the field.
Add an optional talos.imageFactoryCA taking the CA in PEM form. When set,
the chart renders it into a ConfigMap in the release namespace — CDI
requires certConfigMap to sit alongside the DataVolume — and references it
from the worker DataVolume on the HTTP import path; a pool cloning a
golden image via osImage.builtin builds no source.http and renders
neither. When empty, nothing is rendered and the output is byte-for-byte
what it was.
CDI appends the bundle to the system pool rather than replacing it, so a
release pulling from a publicly-trusted endpoint keeps verifying either
way.
The value is emitted verbatim into a ConfigMap readable by anything that
can read the namespace, so the render fails closed on private key material
and on anything that is not complete PEM certificate blocks. Those guards
live in a new validate-only cozy-lib.tls.validateCACert helper that takes
the caller's field name for its messages, and cozy-lib.tls.caCertSecret is
refactored onto it, so the check exists in exactly one place.
Assisted-by: LLM
Signed-off-by: Fedor Batonogov <fedor@batonogov.ru>
cozy-lib.resources.sanitize divides the CPU limit by cpuAllocationRatio
and rendered the quotient as a bare float. When the ratio does not
divide the value evenly the result is finer than a milli-CPU, and the
VPA admission webhook rejects it outright:
admission webhook "vpa.k8s.io" denied the request:
minAllowed: CPU [62500u] must be a whole number of milli CPUs
packages/extra/etcd routes its VPA minAllowed/maxAllowed through the
helper, so 250m / 4 = 62.5m left the etcd HelmRelease looping
UpgradeFailed -> RollbackFailed forever: the rollback target renders the
same invalid value. Since v1.6.0 HelmReleases wait on each other, so
that one stuck app also blocks tenant-root, cozystack-basics and
kubevirt-cdi-operator, and without CDI the Talos worker rollover cannot
import the worker OS image. Only the divisors of 250 worked as ratios,
which is an artefact of one chart's hardcoded floor rather than a real
constraint on the operator.
The request is rounded up, never down, so it stays at or above the
intended fraction of the limit. It is rounded only when it has to be:
if the milli-CPU value equals its own ceiling, the float keeps its
current rendering, so nothing changes on the default ratio of 10 or on
any ratio that divides evenly; otherwise it renders as the ceiling in
m. The comparison is exact, with no tolerance. sprig's divf and mulf run
on shopspring/decimal rather than float64, so mulf (divf 0.25 10.0)
1000.0 renders exactly 25 and there is no binary-float noise on this
codepath to absorb. A tolerance would reintroduce the bug: a value
whole-milli only to within it, such as 100.0000004m at ratio 1, would
read as exact and render "0.1000000004", rejected for the same reason
62.5m is. The same predicate covers sub-milli input, since ceil carries
any positive request to at least 1m, and a zero request stays "0".
Fixes#2917
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
The override case injected no _namespace, so it passed through the
absent-map branch and never reached the .Values.host override check.
Inject a non-empty _namespace with an empty host so the case actually
covers the override skip.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add cozy-lib.ns-host-required: it returns the tenant apex like
cozy-lib.ns-host, but fails the render when the platform has injected
_namespace and its host is empty AND the app sets no host override
(.Values.host). Charts that compose a service hostname as <sub>.<apex>
otherwise emit a malformed trailing-dot host silently when the apex is
unset. The guard is scoped to the injected state, so a standalone render
(helm lint / helm template with no _namespace, or an empty _namespace
map) stays renderable. This mirrors the gateway path, which fails on an
empty apex only when a Gateway is actually rendered. Callers use it only
at a hostname-emitting site, so a tenant with no apex and no public
exposure is unaffected.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The negative-floor guard read $min, already int .minReplicaCount, and int
truncates toward zero, so a fractional negative such as -0.5 became 0 and slipped
past into exactly the scale-to-zero-capable ScaledObject the type check above
exists to prevent. Test the untruncated value instead.
The annotations test only set an extra key with no pause, so it exercised the
pass-through branch and never the merge where the pause pair is the destination
and must win a key collision. Pass a colliding paused-scale-in="false" alongside a
distinct key and assert the pause value overrides it on the rendered object.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
cozy-lib.keda.scaledObject only checked max >= min, so an all-negative or
mixed-sign pair (min -5, max -3) passed the ordering guard and rendered a
ScaledObject with a nonsensical floor. Reject minReplicaCount < 0 before the
ordering check. Also add a cozy-lib-tests case for the documented annotations
parameter, which had no caller and no coverage. Follow-up from #3954 (#4187).
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.
Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.
Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.
Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).
Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.
Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.
Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address IvanHunters' deep review round on #3954:
- [MAJOR] Enabling autoscaling directly on an existing non-quorum database no
longer collapses it to 1 instance and deletes every replica PVC. The chart now
SEEDS spec.instances at the quorum floor (effectiveMin) on BOTH the quorum and
non-quorum active paths instead of omitting the field in the non-quorum case;
the constant seed is a three-way-merge no-op so KEDA still owns the runtime
count via /scale, and the enable dip is capped at the floor (>=2) rather than
collapsing to 1. Documents the client-side-3-way-merge (not SSA) assumption
that the no-op relies on.
- [MINOR] Fail closed on autoscaling.minReplicas > autoscaling.maxReplicas instead
of silently resolving the ceiling upward (which defeated maxReplicas as a cost
knob). The stray maxReplicas: 0 case now rejects rather than clamps.
- [MINOR] The disable-wedge guard, its values note and its test no longer emit the
invalid `kubectl scale --subresource=scale` (scale has no --subresource); use
`kubectl scale cluster.postgresql.cnpg.io/<name> --replicas=N`.
- [MINOR] Correct the enablement note: raising minReplicas/maxSyncReplicas on a
live cluster is NOT harmless when KEDA's live count exceeds the new floor (it
decommissions surplus replicas + PVCs); prescribe pin-first for raises too.
- [MINOR] Add DatabaseAutoscalerKedaOperatorDown (absent(keda_build_info)): when
the operator is unhealthy a ScaledObject never becomes an HPA and no other alert
can fire. Intersect DatabaseAutoscalerScaleStuck with a Pending pod so a healthy
long basebackup (Running pod, desired>current >15m) no longer false-fires.
- [MINOR] Keep the KEDA CRDs on package uninstall (crds.additionalAnnotations
resource-policy: keep via the shim) so removing the keda package doesn't
cascade-delete every tenant ScaledObject.
- [MINOR] cozy-lib helper: reject a non-numeric bound (int "two" -> 0 -> a
scale-to-zero ScaledObject) and render an explicit pollingInterval/cooldownPeriod
of 0 (`if not kindIs invalid`, not `with`).
- [MINOR] Note the read-load fail-open: a broken metrics join reads as idle and
silently pins autoscaling at the floor under real load.
New/updated tests cover the non-quorum seed, the min>max guard, the corrected
kubectl message, the operator-down + Pending-intersected alerts, the CRD keep
policy, and the non-numeric-bound helper guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Address the remaining IvanHunters review points on #3954:
- [MAJOR] Disabling autoscaling wedged the HelmRelease under synchronous quorum.
An autoscaled + maxSyncReplicas>=1 cluster runs with instances seeded from
effectiveMin while replicas stays at its default, so flipping autoscaling off
rendered instances: <replicas> next to the quorum and CNPG's webhook rejected
it ("maxSyncReplicas must be lower than the number of instances"). Add a
symmetric render-time guard on the static-count path (covers off/dryRun/
transition): fail with an actionable message when replicas <= maxSyncReplicas.
- [MINOR] Role-scope the freeze brake's currentInstances count
(label_cnpg_io_instance_role=~".+") so lingering CNPG join/bootstrap Job pods
(cnpg.io/cluster but jobRole, not instanceRole) don't inflate it and nudge
desired up while the brake is meant to hold it.
- [MINOR] Default maxReplicationLagSeconds to 0 (brake opt-in): only the
non-braking path is validated live, so ship the validated default and let the
freeze be enabled explicitly until its branch is exercised under load.
- [MINOR] Document the floor-change and non-quorum-disable dips (same three-way-
merge hazard) and the pin-replicas-first mitigation.
- [NIT] Coerce the cozy-lib helper's `paused` (eq "true") so a caller passing the
string "false" does not pause the ScaledObject.
- [NIT] Trim the keda.clusterDomain injection comment so it no longer ships ~8
verbatim lines into the cozystack-values Secret on every cluster.
Regenerate types/schema/README/postgres-rd. New tests cover the disable-wedge
guard, the role-scoped freeze count, and the string-"false" paused coercion.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Close the last uncovered rejection path of cozy-lib.keda.scaledObject: a
non-map scaleTargetRef now has a failedTemplate assertion, so all twelve
fail-closed guards are executable-documented.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Complete the helper's rejection-path coverage: add the two remaining
fail-closed assertions (minReplicaCount / maxReplicaCount required) so all
eleven guards are executable-documented.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The helper validates ~10 required inputs but the suite only exercised three
rejection paths (non-dict arg, missing scaleTargetRef.name, inverted bounds).
Add a parameterised fixture that builds a fully-valid argument dict and drops
exactly one required field per case, and assert the fail message for name,
scaleTargetRef.kind, scaleTargetRef.apiVersion, query, threshold and
serverAddress. The helper's rejection contract is now executable-documented in
full rather than sampled.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Two review findings:
- The enablement note claimed the two-phase `transition` flow avoids the
count dip when enabling autoscaling on an existing cluster. It does not:
helm-controller upgrades with Helm's three-way merge (not SSA), which keys off
the old-vs-new manifest diff, not KEDA's live /scale value. At phase 2 the
chart stops rendering the staged static count, so the field is deleted (no
quorum -> CNPG defaults to 1, a brief collapse) or changed (quorum -> rebased
to effectiveMin), then the warm KEDA restores it. Rewrite the docs to state the
flow reduces (KEDA is pre-warmed, no cold-start) but does not eliminate the dip,
and spell out the suspend + scale-subresource pin that actually makes the
handoff dip-free. This is the SSA-vs-three-way-merge handoff the design proposal
left for the PoC to settle.
- Pin metricType: AverageValue on the Prometheus trigger in
cozy-lib.keda.scaledObject. The desired = ceil(query / threshold) identity
holds only for an AverageValue external metric; it was relying on KEDA's
default. Make it explicit and assert it so a re-vendor or default change cannot
silently break the arithmetic.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Implement the KEDA-based read-replica autoscaler for postgres per the
merged design proposal (cozystack/community#44):
- cozy-lib.keda.scaledObject: engine-agnostic helper rendering a KEDA
ScaledObject (scale target, effective bounds, prometheus trigger,
scale-down behavior, optional pause), fail-closed on bad input.
- postgres: an `autoscaling` values block (enabled / minReplicas /
maxReplicas / target / maxReplicationLagSeconds / dryRun). Under active
autoscaling the chart drops the static .spec.instances so Flux and the
KEDA-managed HPA no longer contend, and renders a ScaledObject whose
minReplicaCount folds in the synchronous-quorum floor
(max(minReplicas, maxSyncReplicas+1, 2)); when the floor exceeds
maxReplicas, quorum wins. dry-run keeps the static count and pauses the
ScaledObject. The WorkloadMonitor tracks the quorum floor instead of a
stale static value.
Covered by helm unittest (cozy-lib helper: valid render, pause, three
fail-closed guards; postgres: instances dropped, quorum-floored bounds,
quorum-over-max, dry-run pause). Regenerated values.schema.json, README,
Go types and postgres-rd.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
cozy-lib.resources.toFloat knew every suffix of the Kubernetes quantity
grammar except `u` and `n`. A suffix it does not recognise is not an
error: the unit letter stays on the string, `float64` cannot parse the
result and yields zero silently, and the caller divides or compares
that zero as if it were the value.
The node group CPU request guard is one such caller. A request of 100u
above a limit of 50u compared zero against zero and rendered, so the
pair reached API server validation on the worker's Pod instead of
failing where the values were written.
Pinned by a suite over the whole suffix grammar, one value per form,
because the failure has no error to notice it by.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
`_cluster` and `_namespace` reach a chart through valuesFrom on the
cozystack-values Secret, so they are absent whenever a chart is rendered
on its own: `helm lint`, a plain `helm template`, and any helm-unittest
suite that does not set them. Twelve of the helpers in _cozyconfig.tpl
read straight through the missing map and abort the render rather than
returning their documented empty default.
Two spellings, one cause. `index .Values._cluster "root-host"` raises
`index of untyped nil`, and `.Values._namespace.host` raises
`nil pointer evaluating interface {}.host`. In both cases the trailing
`| default ""` never runs: it applies to the result, and there is no
result, only an error. That default covered a key missing from a map
that exists, never a missing map.
Guard the map instead, which is what `cozy-lib.images-registry` in this
same file already did:
(index (.Values._cluster | default dict) "root-host") | default ""
`index` on an empty map returns nil harmlessly, so the trailing default
applies as it always read. With the map present, `| default dict`
returns it untouched.
This changes no rendered output. Every one of the 34 charts that symlink
cozy-lib was rendered before and after, both with the injected values
populated and with them set to empty maps, and every result is
byte-identical. Eight charts differ between any two runs because they
generate passwords; rendering the unchanged tree twice reproduces
exactly those eight, which is how they are excluded.
What this does not do is fix a chart that fails today. The charts that
cannot render standalone read the injected values directly in their own
templates rather than through these helpers, so they are unaffected by a
change to the library and are being addressed where they live.
Twelve of the thirteen helpers here have no callers anywhere in
packages/ at present; only `cozy-lib.images-registry` is reached, from
`cozy-lib.image`, and it was already guarded. The guard is still worth
having on the rest: they are the library's public surface, they are what
a chart author copies when reaching for cluster configuration, and the
broken spelling propagates by being read. `cozy-lib.image` is called
from four templates, so these helpers do get adopted once they exist.
Add a suite in cozy-lib-tests, which is where library behaviour is
pinned because a library chart renders nothing of its own. One case
covers the uninjected state, which needs no mocking since neither key is
in values.yaml, and one covers the injected state so the guard cannot
quietly change what a populated map returns. The injected case includes
`scheduling` and `branding` specifically: those two are the helpers
whose body was restructured rather than wrapped, so they are the ones
whose present-map behaviour most needs pinning. Against the unguarded
helpers the first case fails with `index of untyped nil`.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rename the Secret cozy-lib.tls.caCertSecret renders from "<release>-ca-cert"
to "<release>.tenant-ca", the same canonical, key-free name the
CA-extraction controller publishes, so a chart that holds its CA PEM at
render time and an operator whose CA is minted asynchronously converge on
one object a tenant can name. "<release>-ca-cert" is not a free name —
Percona Server for MongoDB already claims it with a key-bearing Secret.
Harden the write guard: require a complete BEGIN/END CERTIFICATE block,
keep failing closed on any PEM private-key header, and coerce a non-string
caCert value before trimming so an unquoted numeric YAML scalar cannot
panic the template.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Per-app TLS issues a per-release self-signed CA, but the only objects
that carry ca.crt — the cert-manager CA and leaf Secrets — also carry
private keys. Granting a tenant read on those to obtain the trust
anchor also hands over a server or CA private key.
Add cozy-lib.tls.caCertSecret, which renders a canonical Opaque Secret
holding only ca.crt and labelled internal.cozystack.io/tenantresource=
true. The tenantsecret registry surfaces label-bearing Secrets under
core.cozystack.io/tenantsecrets, which the base tenant roles already
grant, so tenants reach the trust anchor without any grant on a Secret
that contains tls.key. The helper fails closed on empty input and
refuses any PEM carrying private key material.
Covered by helm-unittest in the cozy-lib test chart.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This patch breaks introduces a helper function in cozy-lib to correctly
handle special case resources when transforming a nested map of limits
and requests to a flat map suitable for use in resourceQuotas. As a
result, admins can now specify any types of resources as resource quotas
for tenants, and they will be correctly transformed to the correct
format for the underlying kubernetes ResourceQuota. In addition to the
previously supported compute resources, such as CPU, memory, and custom
resources, like GPUs, special quota strings such as
"services.loadbalancers" are now correctly handled.
```release-note
[cozy-lib,platform] Support resource quotas for special kubernetes
quotas, such as service.loadbalncer count and others.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
## What this PR does
The cozy-lib library package got complicated enough to warrant its own
unit tests. Since unit tests are a "good thing" (tm), a somewhat generic
framework for running all kinds of unit tests was introduced into the CI
pipeline and Makefile targets. For now all it runs is `make test`
against the `packages/{library,apps,system,extra}/*` directories,
wherever a `test` target is present in the Makefile, and for now this is
only for the `cozy-lib` Helm library chart.
### Release note
```release-note
[ci,dx] Introduce a scaffold for running unit tests locally and in CI
and add the first unit tests for the cozy-lib helper Helm chart.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Fedor Batonogov