## What this PR does
Two fixes for the optional KMS proxy of the platform Keycloak
(`encryption.*`).
The default static backend never started. The chart stored 32
alphanumeric characters as the KEK, but the proxy wants base64 of 32
bytes and exits with `config: KKP_KEK must decode to 32 bytes (got 24)`.
The KEK is now generated with `randBytes`, and a stored key of the old
form is replaced. The proxy never ran with such a key, so nothing was
encrypted under it. Any other stored key is kept if the proxy accepts it
and fails the render otherwise, so the chart never overwrites key
material. The KEK Secret now carries `helm.sh/resource-policy: keep`:
the DEK set is wrapped with it, so it has to survive turning encryption
off or reinstalling the package.
Without `deksetSecretName` the proxy mints a new DEK set on every start
(`buildCipher` in `cmd/proxy/main.go` of keycloak-kms-proxy) and keeps
it only in memory. After the first restart of the proxy pod it cannot
decrypt anything it wrote before. The values comment said this is safe
on a fresh DB, it is not. The chart now refuses to render the proxy
without a DEK set unless `fields: disabled`. Passthrough still works
without one, so the static backend can get its KEK Secret created before
the DEK set is minted with it.
Breaking: a release with encryption on and no `deksetSecretName` fails
to render on upgrade until a DEK set is configured. Such a setup loses
its data on the next proxy restart anyway, so failing loudly is the
point. A stored static KEK the proxy cannot decode fails the render too,
instead of being replaced.
Checked with `helm unittest` (100 tests, the new ones fail on the old
template) and by starting keycloak-kms-proxy v0.2.3 with the KEK the
chart renders now. With the KEK from the old template it fails as quoted
above.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/742 documents the
setup with a DEK set
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
The other repositories do not touch the keycloak chart values.
### Release note
```release-note
fix(keycloak)!: The KMS proxy of the platform Keycloak now requires `encryption.deksetSecretName` unless `encryption.fields` is `disabled`. Without it the proxy generated a new data key on every start and could not read data encrypted before a restart. The static backend now generates a KEK the proxy accepts; before, the proxy could not start with it. The KEK Secret is no longer deleted when encryption is turned off or the package is removed.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Configuration**
* Enabling field encryption requires a persistent DEK set. Chart
rendering reports an error if it is missing; setting fields to
`disabled` does not require one.
* For the static backend, a valid stored KEK is reused, and a missing or
legacy-format KEK is replaced. An unusable stored KEK causes rendering
to fail.
* The KEK Secret is retained when encryption is disabled or the package
is removed.
* **Documentation**
* Clarified DEK set requirements and sharing across starts and replicas,
and updated guidance on configuring multiple proxy replicas.
* Warns against deleting the retained KEK while encrypted values remain.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Boundary cases must catch removal of the constraints and the error at
253 characters.
Assisted-by: LLM
Signed-off-by: Федор Батоногов <f.batonogov@yandex.ru>
The system truststore also verifies HTTPS client certificates unless a
separate HTTPS truststore is configured. Operators adding an LDAP CA
need to know that this can extend inbound TLS trust as well.
Assisted-by: LLM
Signed-off-by: Федор Батоногов <f.batonogov@yandex.ru>
Private LDAPS, identity-provider and SMTP endpoints need their CA in the
Keycloak truststore. Environment variables alone cannot supply that file,
and manual StatefulSet patches are overwritten by Flux reconciliation.
Assisted-by: LLM
Signed-off-by: Федор Батоногов <f.batonogov@yandex.ru>
Any stored KEK that did not decode to exactly 44 characters was
overwritten. The proxy skips CR and LF when it decodes KKP_KEK, so a
key restored from a file ending in a newline works, and the chart
replaced it silently. A 44-character value that is not base64 of 32
bytes was kept, and the proxy could not start with it.
Only the 32-character alphanumeric form earlier chart versions wrote
is replaced now. A key the proxy accepts is kept, and anything else
fails the render, so key material is never overwritten.
Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The chart now requires a persistent DEK set wrapped with the static
KEK, so the KEK Secret has to outlive the release. Disabling
encryption or reinstalling the package deleted it, and the next
enable minted a fresh KEK that cannot unwrap the DEK set. The lookup
already reuses a kept Secret on the next install.
Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Without encryption.deksetSecretName the proxy mints a new DEK set on
every start and keeps it only in memory. After any restart of the
proxy pod it can no longer decrypt the rows it wrote before, so the
default configuration loses access to all encrypted user data on the
first rollout, drain or OOM.
Refuse to render the proxy without a DEK set unless encryption.fields
is "disabled". Passthrough keeps a way for the static backend to get
its KEK Secret created before the DEK set is minted with it.
Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The proxy reads KKP_KEK as base64 of 32 random bytes and refuses to
start otherwise. The chart stored 32 alphanumeric characters, which
decode to 24 bytes, so the default static backend never started:
config: KKP_KEK must decode to 32 bytes (got 24)
Generate the key with randBytes and replace a stored key of the old
form. The proxy never ran with such a key, so nothing was encrypted
under it.
Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The Keycloak StatefulSet and the database proxy can already be spread
across failure domains, but the CloudNativePG Cluster behind them could
only follow the cluster-wide topology constraint, with no way to pick
zones for this database alone. With both instances in one zone, losing
that zone loses the database and every login with it.
Add database.affinity (CloudNativePG shape) and
database.topologySpreadConstraints, with empty defaults so nothing
changes unless an installation opts in. The cnpg.io/cluster label
selector is added to constraints that carry none, and the constraints
replace the cluster-wide ones for this database when set.
Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Neither the Keycloak StatefulSet nor the keycloak-kms-proxy Deployment
exposed any placement, so the scheduler was free to put both replicas of
each in one failure domain. For Keycloak that is worse than it looks: a
StatefulSet does not replace a pod stuck on an unreachable node, so losing
that domain takes every replica for as long as it stays lost.
Expose affinity and topologySpreadConstraints for both, empty by default
so nothing changes unless an installation opts in, and give the proxy a
PodDisruptionBudget once it runs more than one replica.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
0.2.3 publishes the proxy as a linux/amd64 and linux/arm64 index;
earlier releases were amd64 only.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The list was read only inside the adminHost branch, so a value the chart
would refuse — master, a scalar, a name that splits the path — sat in the
values unnoticed until someone turned the admin hostname on, and the refusal
then landed on the change of authentication topology rather than on the edit
that caused it. Build the list on every render and let the branches use the
result.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The over-budget message derived the realm count from the path list, which
also carries the two static paths, so it named one realm more than the
deployment has — the one number an operator acts on. Keep the realm list in
its own template and count it there, and derive the budget from the static
paths rather than restating 31.
The shape guard also let a falsy scalar through, so `exposedRealms: false`
reached range and failed with a Go template error instead of the message that
says what to write.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
A Gateway API route rule takes 64 path matches and each published realm takes
two of them, so past 31 realms the apiserver rejects the whole HTTPRoute and
the login hostname stops answering. Fail while rendering, naming the
parameter to shorten, instead of leaving the limit in a comment. The Ingress
path has no such cap, so the check lives with the route.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
exposedRealms replaced the published realm instead of adding to it, so a
deployment naming only its back-office realm took its own login off the
public route: every component's issuer URL points at the platform realm, and
they would all have met a 404. Publish the platform realm unconditionally and
deduplicate, which is also what the parameter's description promised.
Refuse master in either position rather than warning about it in a comment —
publishing it is exactly what the realm enumeration exists to prevent.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
ingress-nginx renders `pathType: Prefix` as a plain nginx prefix location
instead of the element-wise match the Ingress API specifies, so a lone
/realms/cozy also served /realms/cozy-admin — a realm the chart claims is
unpublished. Publish each realm as an exact path plus a trailing-slash
prefix, which bounds the match on every controller and on Gateway API.
While here, reject a scalar in place of the realm list instead of failing
inside range, and point a malformed platform realm name at the parameter
that carries it rather than at exposedRealms.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The narrowed login route carried a bare /realms prefix, so it also published
realm master: its token endpoint accepted super-admin credentials from the
internet. Enumerate the published realms instead — the platform realm by
default, ingress.exposedRealms when a deployment serves more than one.
Behavior changes only where ingress.adminHost is already set.
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The Keycloak comment and the helper doc claimed the tenant LimitRange as the
reason for the resources, but cozy-keycloak has no LimitRange and neither
does a tenant that leaves resourceQuotas empty: there the sidecar had no
requests and no limit at all, and the change gives it a 1Gi ceiling where
there was none. The comments and both doc blocks now say what applies to
each caller. No rendered object changes.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
The barman-cloud plugin injects its sidecar without resource requests or
limits. Tenant namespaces ship a LimitRange that defaults containers to
128Mi, so the sidecar inherits that default and is OOMKilled mid-backup:
the ObjectStore stays healthy, the Cluster stays Ready, and only the
backup fails. A cgroup high-water mark of 254 MiB on a 38 MB database
shows the default cannot hold it.
The shared cozy-lib helper and the Go path that builds the platform's own
ObjectStore now request 100m/256Mi and cap memory at 1Gi. No CPU limit is
set, matching the entityOperator precedent in the kafka chart: a
throttled sidecar stalls WAL archiving instead of failing it.
The helper is renamed from checksumSidecarConfiguration to
sidecarConfiguration, since it no longer carries only the checksum pin.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Ten system charts cannot be rendered on their own. They read the
platform-injected _cluster channel, and two of them (bucket,
monitoring) also read _namespace, without declaring either key in
values.yaml. On a cluster the platform supplies both through the
cozystack-values Secret, so installs are unaffected, but rendering a
chart by itself dies before a single template is emitted: on "index of
untyped nil" for a missing _cluster, and on "nil pointer evaluating
interface {}.host" for a missing _namespace.
Declare the keys with empty defaults. Helm merges chart defaults under
the supplied values, so an injected _cluster still wins and rendered
output does not change. Each of the ten also gets a suite that renders
with nothing injected; drop a declaration again and that chart's suite
goes red.
This makes `helm template` succeed for all ten. `helm lint` also
reports a missing cozy-lib dependency on keycloak and monitoring,
which is a separate pre-existing defect and survives this change.
etcd-operator reads _cluster too and stays undeclared here: its single
call site already guards with `| default dict`, so the chart renders
standalone as it is.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
SSL_CERT_FILE alone does not narrow trust. crypto/x509 replaces only its file
list and then scans SSL_CERT_DIR unconditionally, and the pinned proxy image
ships /etc/ssl/certs/ca-certificates.crt, so the connection guarding the KEK
trusted the private CA plus every public root — a mis-issued certificate for the
Vault hostname was still accepted. Set SSL_CERT_DIR to the mount as well, which
collapses the pool to the CA the operator configured.
An unvalidated caBundle failed open on top of that: a non-PEM value is dropped
silently by AppendCertsFromPEM, leaving the proxy on the image's roots with no
error at all. Reject it at render time, the way backend and auth already are.
Rotating an inline caBundle had no effect either. The pod template did not
change, so no new pods, and crypto/x509 memoizes the pool per process — the
upgrade reported success while the old CA stayed in use until some later
restart. Derive a checksum/vault-ca annotation from the bundle so the rollout
happens with the change.
Also add vault.caSecretKey: the projected key was hardcoded to ca.crt, and a
bundle stored under another name (a cert-manager Certificate from an ACME issuer
emits no ca.crt at all) wedged the pod at mount time with no way to correct it
from values.
Drop the claim that the database leg verifies against its own CA pool: the chart
never sets KKP_BACKEND_CA_FILE, so that leg is plaintext today, as the note 30
lines above already says. The conclusion holds — it never consults the system
trust store — but for the other reason.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The volume and mount that consume the Vault CA sit at pod level, outside the
vault-transit env block, while $vaultCASecret was seeded from caSecretName
regardless of the backend. With kms.backend=static and caSecretName set, the
Deployment mounted a Secret the chart never renders, so the proxy — the single
choke point for all Keycloak DB traffic — would sit in ContainerCreating.
Gate both CA keys, and the mutual-exclusion check between them, on
vault-transit, so the whole vault.* subtree behaves the same way under static:
ignored. Project only ca.crt from a pre-seeded Secret, which keeps a
cert-manager tls.key out of the container and turns a bundle stored under
another key into a mount failure rather than an opaque TLS error later.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The proxy could only reach a Vault whose certificate chains to a publicly
trusted root, so a Vault fronted by an internal load balancer with a
self-signed certificate was unreachable over HTTPS.
Allow the CA to be supplied inline (caBundle, the chart renders the Secret) or
as a pre-seeded Secret (caSecretName), mounted and exposed as SSL_CERT_FILE.
That redirects only the system trust store, which nothing but the Vault client
consults — the database leg verifies against its own CA pool.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
notContains passes whenever its path does not resolve, and also
whenever it resolves to something that simply lacks the forbidden
content. A different container at the same index qualifies, so these
guards go green the moment a sidecar is placed ahead of the workload
they were written for, while the flag or env var they forbid comes
back on the real container.
Assert the identity at that index first, in each affected case: the
container name for the Deployment-shaped ones, and for the Cilium
policy its selector, since a deny rule has no name. The selector
identifies that rule only while there is one rule to identify, so it
is paired with a length assertion, and with a second one on the
rule's toPorts, because that path carries two indexes and anchoring
the outer one alone leaves the inner free.
The anchor goes only where nothing else holds the index. A positive
contains on the same path fails on its own once a sidecar arrives
ahead of the workload, and an equal on the whole argument list fails
the same way, so neither carries a separate anchor. A negative does,
including the per-element form below: a wildcard resolving to the
sidecar's own arguments matches nothing and passes.
Several of the guards could not fail for the reason they name, and
the anchor alone would have left that hidden behind a comment saying
they were covered. contains and notContains compare elements for
equality, so a mapping is compared whole: `{name: GF_...}` never
matched an entry carrying a value beside the name, and `{port: 7473}`
would not match that port re-added with a protocol beside it, which
is how the neighbouring entry in the same list is written. Those get
`any: true`.
Equality against a rendered argument is the same failure worn
differently, and it arrives in three disguises. A prefix that never
equals an element: `--pinned-tenants=` never equalled
`--pinned-tenants=tenant-a=shard0`, and `--target-host=` never
equalled a rendered `--target-host=<value>`. One value out of many
under a name that claims the flag: `--compatibility=cilium` left the
kilo default render free to carry `--compatibility=flannel` off the
same values key, and the two `--proxy-fqdn` negatives forbade only
the FQDN the default cluster-domain produces. And a form that holds
by luck: `--gateway-api` still fails today only because that template
renders it bare, so an upstream switch to `--gateway-api=true` would
retire the guard in silence.
All of those become notMatchRegex over args[*], which matches the
flag name per element and fails on any value and on the bare form
alike. A length assertion would not have covered the prefix cases,
since an upstream bump that adds the flag while consolidating another
argument keeps the count. A wildcard that resolves to nothing still
passes, so the index anchors stay.
The whole argument list is a different instrument, answering what the
default invocation is rather than whether one flag is absent. It
fails on an argument added, removed or swapped instead of only on the
ones someone thought to enumerate, which is worth having where the
template is first-party and a failure is a deliberate change asking
to be read. flux-shard-operator gets it in `renders the default
Deployment shape`, replacing eleven individual contains that listed
exactly the same eleven elements; `--shard-cpu-limit=` was dead for
the prefix reason and is dropped rather than repaired, because that
list now covers it. The vendored ouroboros templates get no such pin:
upstream churn arrives there on its own schedule, and a whole-list
assertion would fail a test named for one flag for a reason that has
nothing to do with that flag.
Anchor every index on a path, not the outermost one. The Cilium
assertion reads spec.ingressDeny[0].toPorts[0].ports, and pinning the
rule list alone would still let the port return on a second toPorts
entry of the same rule.
Each guard was checked by rendering what it forbids onto the element
it names, and each anchor by rendering the shift it exists to catch.
The form is already used in this tree, in opensearch-operator and
dashboard. Assertions of the same shape remain in files this commit
does not touch, including one in a package it does, and stay out of
scope here.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
helm-unittest coverage for the S3 request-checksum fix (was mutation-negative
before): assert spec.instanceSidecarConfiguration.env pins
AWS_REQUEST_CHECKSUM_CALCULATION=when_required on all rendered barman-cloud
ObjectStores — postgres backup + bootstrap-recovery
(packages/apps/postgres/tests/backup_storage_test.yaml) and the Keycloak DB
(new packages/system/keycloak/tests/db_backup_test.yaml). Verified by mutation:
reverting the template change now fails these assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Address review on the S3 request-checksum fix:
- Extend the fix to the platform-managed useSystemBucket=true flow, whose
ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
barmanSidecarConfiguration(). The platform's own default system bucket is
SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.
- Factor the duplicated instanceSidecarConfiguration block (3x across two
charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
The barman-cloud plugin's sidecar uploads via boto3. Since botocore ~1.36
(early 2025) the default RequestChecksumCalculation is when_supported, which
attaches a flexible checksum to every PutObject. Non-AWS S3-compatible
backends (Ceph RADOS Gateway, some MinIO / Cloudflare R2 builds) reject the
accompanying header with "InvalidArgument: x-amz-content-sha256 must be
UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256
value", so every backup/WAL-archive upload fails against them.
Pin the sidecar to when_required via the ObjectStore's
spec.instanceSidecarConfiguration.env (botocore honors the
AWS_REQUEST_CHECKSUM_CALCULATION env var) on both the keycloak system DB and
the postgres app (backup + recovery ObjectStores). when_required is also
accepted by AWS S3 on a plain PutObject, so it is a safe default everywhere.
This mirrors the etcd-operator fix (cozystack/etcd-operator#342, released in
v0.5.3), which applied the same WhenRequired policy to its aws-sdk-go-v2 S3
client for the same Ceph RGW backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
Fixes#3300. Migrates CNPG Postgres backups off the deprecated native
`spec.backup.barmanObjectStore` (removed in CNPG 1.29; the
`barman-cloud-*` binaries it needs are absent from the `standard` image
variant keycloak-db pins, and the bare system-flavor tags apps/postgres
pins are deprecated upstream) onto the [CloudNativePG Barman Cloud
plugin](https://github.com/cloudnative-pg/plugin-barman-cloud), across
every Postgres backup surface: `apps/postgres`, `system/keycloak`, and
the `cozy-default-cnpg` BackupClass strategy driver. The chart renders
`spec.plugins` + a `barmancloud.cnpg.io/ObjectStore` CR; the
backup-controller driver SSA-applies an `ObjectStore` and patches
`spec.plugins`; `Backup`/`ScheduledBackup` use `method: plugin`. The
`postgres-operator` chart vendors and co-locates the
`plugin-barman-cloud` operator (with its cert-manager-issued mTLS) so
CNPG can discover it.
One platform-level change is required for the plugin to run and ships
with this PR (see the upgrade notes in
`docs/operations/backup-classes.md` and the release note below): the
CloudNativePG operator is bumped to 1.28.1 (also fixing an operator/CRD
version skew that broke all backups). The other prerequisite — the
LINSTOR scheduler admission webhook stripping
`initContainers[].restartPolicy` cluster-wide, which broke the plugin's
restartable sidecar — is already fixed on `main` (f4aef7305 re-vendors
`linstor-scheduler` to upstream chart 0.3.1 / extender v0.3.6, which
keeps the webhook and stops stripping unknown pod fields), so after the
rebase this PR no longer touches `linstor-scheduler`; just do not hold
that package back on an older version when rolling out the plugin.
Supersedes #3306 (the keycloak `system`-image workaround): keycloak-db
now backs up via the plugin on the `standard` image, so that stopgap is
no longer needed.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: docs/operations/backup-classes.md changed; a website mirror
follow-up is warranted (not yet opened).
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(backups): migrate CNPG Postgres backups to the CloudNativePG Barman Cloud plugin
Postgres backups (apps/postgres, system/keycloak, and the cozy-default-cnpg BackupClass strategy) now use the CloudNativePG Barman Cloud plugin (spec.plugins + a barmancloud.cnpg.io/ObjectStore, method: plugin) instead of the deprecated native spec.backup.barmanObjectStore (removed in CNPG 1.29; the barman-cloud-* binaries it needs are absent from the standard image variant keycloak-db pins, and the bare system-flavor tags apps/postgres pins are deprecated upstream). The barman-cloud plugin operator is installed alongside the CNPG operator.
One platform-level change rides along and requires operator attention on upgrade:
- The CloudNativePG operator is bumped to 1.28.1 (fleet-wide minor upgrade); this also fixes an operator-newer-than-its-CRDs skew that made the operator report "instance manager was restarted during backup" and fail every backup. The vendored chart ships the CNPG CRDs as ordinary release manifests (crds.create), so helm upgrade updates them with the operator; the skew came from the image.tag pin outrunning the chart's CRDs, which this PR removes.
- The plugin's restartable sidecar requires the linstor-scheduler admission webhook that no longer strips initContainers[].restartPolicy (upstream chart 0.3.1 / extender v0.3.6, shipped separately on main); do not hold the linstor-scheduler package back on an older version when rolling out the plugin.
```
Close#3300, #3246
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Switched PostgreSQL CNPG backups/restores to the plugin-based Barman
Cloud workflow, using dedicated ObjectStore resources for backup and
recovery.
* Added the Barman Cloud CNPG-I plugin chart (deployment, TLS, RBAC,
leader election).
* **Bug Fixes**
* Improved backup/restore attachment messaging when the CNPG Cluster
isn’t ready yet.
* Preserves server-name behavior to avoid WAL-archive prefix issues;
clarified endpoint CA handling.
* **Changes**
* Helm integration no longer uses the legacy
`spec.backup.barmanObjectStore` path; updated gating, credential
projection, and ObjectStore-related RBAC/cleanup.
* Removed the linstor-scheduler admission webhook; updated
CloudNativePG/operator versions and CRDs.
* **Tests**
* Updated controller and Helm-unittest checks for the plugin flow;
added/extended end-to-end backup/restore coverage.
* **Documentation**
* Expanded upgrade/migration and backup documentation; updated backup
example materials and comments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The keycloak-http Service only exposes port 8080, but every HTTPRoute
backendRef referenced port 80. The Gateway API resolves backendRefs
against the Service's declared ports, so the route had no valid backend
and envoy answered 503 for all Keycloak hostnames whenever
gateway-enabled is true. The Ingress template is unaffected: it
references the port by name (http).
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:
Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.
Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Review follow-up (cozystack#3300, non-blocking): the barman-cloud ObjectStore CRD
validates spec.retentionPolicy against ^[1-9][0-9]*[dwm]$, so an operator who
explicitly sets backup.retentionPolicy="" would render retentionPolicy: "" and
the API server would reject the ObjectStore, breaking backup provisioning. Guard
the field in the apps/postgres and keycloak chart ObjectStores with
{{- if .Values.backup.retentionPolicy }} (the Go driver path is already safe via
omitempty). Add a helm-unittest asserting an empty retentionPolicy is omitted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
The keycloak-db backup added in #3174 uses the in-tree native
spec.backup.barmanObjectStore, whose barman-cloud-* binaries ship only in the
CNPG system image variant; keycloak-db is pinned to the standard variant, so
base backups fail with 'barman-cloud-backup: executable file not found'. The
proposed workaround (#3306) switches keycloak-db to the deprecated system image
variant when backups are enabled.
Migrate keycloak-db to the barman-cloud plugin instead, matching the rest of
this PR: render spec.plugins referencing a barmancloud.cnpg.io ObjectStore (the
S3/barman config moves there) and a method=plugin ScheduledBackup, keeping the
Cluster on the standard image (the plugin runs the barman tooling in a sidecar).
This removes the need for the system-variant workaround entirely, so #3306 can
be closed. cozystack#3300.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
## What this PR does
Adds optional scheduled S3 backups for the `keycloak-db` CloudNativePG
cluster.
When `backup.enabled=true` with a `backup.destinationPath`, the Cluster
gets
`spec.backup.barmanObjectStore` (gzip-compressed data + WAL streams, a
configurable `retentionPolicy`) and a `ScheduledBackup` drives recurring
base
backups. S3 credentials are either inlined into a rendered
`keycloak-db-backup-s3-creds` Secret, or referenced via
`backup.existingSecretName` (a Secret carrying `AWS_ACCESS_KEY_ID` /
`AWS_SECRET_ACCESS_KEY`).
Disabled by default — no change to existing deployments; the object
store
coordinates and credentials are supplied per-environment by the
operator.
This is the Keycloak/Postgres half of backing up a Cozystack portal
control
plane. The portal-side etcd backups (system Talos etcd + api-server
etcd) live
in the companion downstream PR aenix-org/cozyportal#1008.
### Release note
```release-note
feat(keycloak): add optional S3 backups (CNPG Barman) for the Keycloak database, configurable via `backup.*` values and disabled by default.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable scheduled backups for the Keycloak CNPG database to
an S3-compatible object store, including destination prefix, optional
endpoint URL, retention policy, and cron schedule.
* Backups and scheduled backup resources are only created when all
required backup settings are provided, avoiding incomplete
configuration.
* Supports backup credentials via inline values (creates the needed
Secret automatically) or via an existing Secret reference.
* **Documentation**
* Added guidance on the cron schedule format required for the scheduler.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The release workflow publishes the OCI tag as 0.2.0 (docker/metadata-action
strips the leading v from the git tag), so the chart default of v0.2.0 does
not exist and pulls fail. Pin the real tag.
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assert KKP_DEKSET_FILE honors a non-default deksetSecretKey, closing the
last uncovered branch of the dekset wiring.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Wire encryption.deksetSecretName -> KKP_DEKSET_FILE (mounted read-only) and
make encryption.replicas configurable. Without a dekset the proxy mints an
ephemeral DEK, which only works on a fresh database and a single replica.
A shared wrapped DEK set is required to migrate an existing populated
Keycloak (the proxy must reuse the DEK the backfill wrapped rows with) and
to run more than one replica; a guard rejects replicas>1 without one.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
v0.2.0 is the first release carrying the Kubernetes/AppRole auth modes
this chart wires, so encryption.kms.vault.auth is now usable with the
default image.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Lock in the SA-object token toggle (off by default, on only for kubernetes
auth), complementing the pod-level assertions.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Address review: the proxy pod ran under the namespace default ServiceAccount,
which the keycloak StatefulSet also uses. Binding a Vault Kubernetes-auth role
to default would hand the keycloak pod itself Transit access, defeating the
isolation the proxy exists to provide. Add a dedicated keycloak-kms-proxy
ServiceAccount, run the proxy pod under it, and mount its token only in the
kubernetes auth mode (off otherwise, to minimise the credential surface).
Docs name the SA to bind the Vault role to.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Extend encryption.kms.vault.auth with "kubernetes" (recommended): the
proxy logs in with its own ServiceAccount token, so no credential is
stored in a Secret. "approle" and "token" remain. Render-time
validation requires kubernetes.role and rejects unknown methods;
helm-unittest covers the new path and the mutual absence of the other
modes' env vars.
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The notContains checks matched whole array elements, so a name-only
content never equaled the valueFrom entry and the assertion could never
fail. Add any:true so "AppRole stores no static token" and, symmetrically,
"token mode renders no AppRole vars" are actually verified (confirmed by
mutation-testing the template).
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
The keycloak-kms-proxy Vault Transit backend only supported a static
token. Add an encryption.kms.vault.auth switch — "token" (default,
unchanged) or "approle" — wiring KKP_VAULT_ROLE_ID / KKP_VAULT_SECRET_ID
/ KKP_VAULT_APPROLE_MOUNT from a Secret. With AppRole the proxy logs in
and re-authenticates on demand, so no long-lived token is stored.
Requires a keycloak-kms-proxy image with AppRole support (> v0.1.0).
Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Address review: CNPG ScheduledBackup.spec.schedule is a 6-field cron
(leading seconds field), so the 5-field default was invalid — switch it to
"0 0 2 * * *". Also gate the ScheduledBackup on backup.destinationPath to
match db.yaml/backup-secret.yaml: schedule has a non-empty default, so the
old guard rendered a ScheduledBackup against a Cluster with no object store
whenever backups were enabled without a destinationPath.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Wire optional scheduled S3 backups for the keycloak-db CNPG cluster via
Barman. When backup.enabled=true with a destinationPath, the Cluster gets
spec.backup.barmanObjectStore (gzip-compressed data + WAL, configurable
retention) and a ScheduledBackup drives recurring base backups.
Credentials are inlined into a Secret or referenced via existingSecretName
(AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY). Disabled by default.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Remove the db.host/db.port/db.urlProperties override and the
extraVolumes/extraVolumeMounts plumbing: they are no longer needed now that
the database endpoint is handled natively. Keycloak's DB host/port come
straight from the keycloak-db-app secret again, except when encryption is
enabled, where the keycloak-kms-proxy owns the repointing. Theme and
admin-console hostname handling are unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Add render-time negative-path coverage for the encryption proxy: an
unsupported kms.backend and a vault-transit backend missing address,
keyName, or tokenSecretName must all fail the template render rather than
silently producing a broken Deployment.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Address review feedback on the encryption values block:
- Spell out the operational caveats an operator must understand before
flipping encryption on: the single-replica proxy turns HA Keycloak into a
single point of failure; existing data is not migrated automatically so
enabling (or later disabling) on a populated database breaks logins until
the offline backfill runs; and a lost static KEK is permanent data loss, so
the KEK Secret must be backed up.
- Add a memory limit to the proxy resources per the repository style guide.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Address review feedback on the encryption proxy template:
- Validate encryption.kms.backend at render time and fail with a clear
message on any value other than static/vault-transit. Both the KEK Secret
and the env wiring now key off the same validated $backend, so a typo can
no longer render a Deployment that mounts a KEK Secret that was never
created (which would silently break the whole IdP).
- Guard the static KEK lookup against a Secret that exists without a usable
'kek' key, preserving the freshly generated KEK instead of nil/empty.
- Default kms.vault to an empty dict and require address/keyName at render
time so an incomplete Vault config fails with an actionable message rather
than a generic nil dereference.
- Publish the metrics port (9090) on the proxy Service so it can be scraped.
- Clarify the backend-TLS note: the proxy negotiates TLS in-band; CNPG CA
pinning is the remaining follow-up.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>