Commit Graph
342 Commits
Author SHA1 Message Date
Alexey Artamonov 27d7dbb140 feat(api): add per-app helm apply-strategy override annotation
Add release.cozystack.io/helm-server-side-apply on the ApplicationDefinition,
threaded through ReleaseConfig into every emitted HelmRelease's
Install/Upgrade.ServerSideApply. Empty leaves the helm-controller default
(server-side apply on v1.5.0); "false"/"disabled" forces client-side apply.

The postgres read-replica autoscaler needs this: KEDA's HPA writes the CNPG
Cluster's /scale subresource (spec.instances) while the chart renders a constant
seed. Under helm-controller's default server-side apply the rendered field is
force-owned and reverts KEDA's live value on every reconcile. Under client-side
apply helm-controller patches the CRD from the previous-vs-new rendered manifest
without consulting live state, so the unchanged constant yields no patch and the
/scale value survives (verified against Helm v4 createPatch: unstructured/CRD
objects take the two-way jsonpatch.CreateMergePatch(original, modified) path
unless threeWayMergeForUnstructured is set, which helm-controller v1.5.0 leaves
at its default).

Guarded by TestConvertApplicationToHelmRelease_ServerSideApplyOverride on the
tenant-release builder (convertApplicationToHelmRelease), and by
TestParseHelmServerSideApplyAnnotation. Removes a stale package_reconciler_test
guard that asserted a false invariant on the wrong (platform Package) builder.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Alexey Artamonov 826f08972f fix(apps/postgres): make autoscaler fail-safe, dip-free, and observable
Address review findings on the KEDA read-replica autoscaler where the
implementation diverged from the merged design proposal.

Fail-safe read load. Gate the `or vector(0)` floor on the existence of ANY CNPG
instance pod (`count(kube_pod_labels{...instance_role=~".+"}) > 0`) — primary or
replica — so the query is empty ONLY when no CNPG pod labels exist at all (a
broken kube-state-metrics pipeline), not during initial provisioning when the
sole replica-side pod is CNPG's join Job (jobRole, not instanceRole). Also pin
`ignoreNullValues: "false"` on the trigger: KEDA's prometheus scaler defaults it
true, turning an EMPTY result into metric 0 (not an error) — the HPA would then
desire 0 and clamp to the floor, i.e. silently scale a broken-pipeline cluster
DOWN. With it false an empty result is a scaler error, so the HPA HOLDS and
DatabaseAutoscalerScalerErrors fires; gating on any instance role keeps a healthy
cluster (including a long basebackup) from tripping that alert.

Constant seed = max(replicas, effectiveMin), so enable/dryRun/transition flips
are a merge no-op with no dip and never undercut the quorum floor.

Fail closed when the quorum floor exceeds maxReplicas (was: effectiveMax silently
raised both bounds, unbounded for a large maxSyncReplicas).

Dry-run / transition keep the HPA (unidirectional pause pair, not the HPA-deleting
full pause) but both SelectPolicy=Disabled, so desired normalizes to current — the
recommendation is read from the read-load metric, not a diverging desired count.

Observability. DatabaseAutoscalerAtMax keys on desired >= maxReplicas, NOT the
ScalingLimited condition (which is also true at the floor, so it fired on every
idle cluster with wrong "raise maxReplicas" advice). Dashboard read-load panels
are per-cluster (label_cnpg_io_cluster) with a CPU panel for ReadCPUUtilization
and the exact clamp in descriptions; ScaleStuck's Pending intersection is scoped
to CNPG pods; alerts.yaml narrative comments moved to Go-template comments so they
do not ship into the PrometheusRule.

Guard the seeding invariant: TestBuildHelmReleaseSpecNoForceOrDriftDetection.
Regenerated README/schema/types.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-04 23:19:25 +03:00
Aleksei Sviridkin 18d02d813a fix(cacert): make the comments match the reconciler (#4064)
## What this PR does

Comments in the CA-extraction package claim guards the code does not
have. No behaviour change; one test is renamed to match the property it
pins.

- `ManagedLabel` said the reconciler only ever writes over a Secret
carrying that label. It does not. `isOurProjection` decides that from
the Secret type and the owner reference, and never looks at the label,
which any writer in the namespace can strip.
- `managedByCozystackLabel` said the reconciler never writes it. It
deletes it whenever the selectors digest changes, which is what gets the
projection re-admitted by the lineage webhook.
- `projectionSuffix` said the dotted canonical name cannot collide with
an operator-owned object. A release the apps API created cannot carry a
dot, since application names are DNS-1035 labels and `release.prefix` is
restricted to `^[a-z0-9-]*$`. The guarantee stops there: the controller
reads the release from the `helm.toolkit.fluxcd.io/name` label, where a
dot is legal, so a hand-written HelmRelease sits outside it. On the
validated path the dot can only come from the suffix an operator or
chart appends, which nothing validates at all, and since a canonical
name ends in `.tenant-ca` and carries that one dot, a suffix ending in
`.tenant-ca` is the only one that reaches a canonical name. The comment
says all of that now.
- The test pinning that name claimed the property for any engine suffix,
in its doc and in its own name, while its loop samples only dot-free
ones. It is now
`TestProjectionNameCannotCollideWithADotFreeEngineSuffix`, and its doc
says what the loop covers.

### Screenshots

Not a UI change.

### Downstream repositories

Comments and one test name in two Go files. No package, values key,
schema field, CRD, metric, release asset or `hack/` target changes, so
nothing in the trigger map fires.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Clarified certificate reconciliation behavior, including label
handling, projection naming, and source certificate scenarios.
* Documented naming guarantees and their API boundaries more precisely.

* **Tests**
* Updated test descriptions and failure messaging to clarify coverage
for dot-free engine suffixes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-04 19:32:01 +04:00
Aleksei Sviridkin 773d8dabce fix(gateway): withdraw TLSRoute acceptance when the mode drops passthrough
Edge terminates TLS at the class provider, so its Gateway carries plain
HTTP listeners only and a route pinned to a passthrough section has
nothing left to attach to. The api, vm-exportproxy and cdi-uploadproxy
endpoints of publishing.exposedServices are the ones that pin one.
Hostname claims are empty for every whole-apex mode, so the status pass
ran over an empty ref set and left the Accepted=True written while the
tenant ran HTTP-01 standing on a route the switch had just unpublished.
Whoever went looking for the dead endpoint read that condition as proof
the route was fine.

The rule this settles on is that the controller owns a condition only
where it judges. Edge judges every attached route by the absence of the
listener it needs and writes Accepted=False with Gateway API's
NoMatchingParent, the reason defined for a parentRef whose port or
sectionName matches no listener. The other whole-apex modes render
those listeners and serve them but collect no claims, so they hold no
opinion: they drop the entry rather than leave a withdrawal standing on
a listener that is serving, naming a class provider that no longer
terminates anything. HTTP-01 collects claims and has already judged
every route this pass could speak about.

A route declaring no hostnames is judged by none of them, because
claims are gathered inside a loop over spec.hostnames and TLSRoute
v1alpha2 sets no minItems on that field. Withdrawing one would write a
condition with no path back, since HTTP-01 renders the listener again
and judges only the routes that produced claims, so both directions
skip it.

The tests read the controller name from the constant the controller
stamps rather than restating it, so a helper cannot go looking for a
name that is no longer written and report absence it never checked.

The switch already deletes the redirect route, the Issuer and the
wildcard Certificate it invalidates. The statuses it owns were the one
thing left behind.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-04 14:33:36 +03:00
Aleksei Sviridkin 5976ddf02f fix(gateway): normalise the GatewayClass name and refuse a blank one
The class lists arrive over the values channel comma-joined, so every
reader trims each entry as it splits them. The scalar those entries are
compared against was not trimmed anywhere. A class name carrying a
stray space therefore failed `has` against its own padded self, and the
two readers of that scalar disagreed about the same class: the gateway
chart put the tenant on an edge-terminated class, while the
cluster-wide ACME issuers still pinned their solver to a listener an
edge Gateway does not render. That stops every HTTP-01 certificate in
the cluster from issuing, with nothing in any status naming the cause.

Normalise once at the single writer so both readers see a clean value,
and keep a defensive trim at each reader. The tenant's own choice does
not travel that channel — it comes from tenant.spec.gatewayClass — so
it is trimmed where the gateway chart reads it.

Unset and whitespace-only are different events and no longer share an
outcome. Unset means the choice was not made, and falling back to the
bundled class stays its documented behaviour. Whitespace-only means a
choice was made and is garbage, in the field that decides where TLS
terminates: resolving it quietly would take a tenant the operator meant
to put behind an edge terminator and publish it directly, with a real
certificate on a real address. Both the platform default and the
tenant's own choice now refuse it by name.

The kind check beside that refusal is part of the same fix, not a tidy
up. Through Helm's values path an unquoted numeric scalar arrives as a
float64, and the whitespace check compares it against a string, so
`className: 123` dies at that comparison — one line before the trim —
with an error naming neither the key nor the file. Refusing whitespace
without checking the kind first would therefore have made the numeric
case worse than it was. `kindIs "string"` is the right test because it
never has to name the type that actually arrived.

The order of the guard's steps is part of the guard. The value is read
before anything normalises it, so a default that fires on any falsy
value cannot launder 0 or false into the empty string and resolve them
to the bundled class ahead of the kind check. Unset is the one case
that still falls back, and each of those four inputs is pinned, so a
normalisation drifting back ahead of a check fails rather than quietly
covering only the loudest bad input.

Also correct a cleanup log line: the wildcard Certificate is deleted
for every mode that is not DNS-01, so naming HTTP-01 in that message is
wrong for existingSecret and for edge alike. And state the listener
name invariant plainly: the cap is 253, not the 63 of a DNS label, and
the longest name the builders produce is 84.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:33:35 +03:00
Aleksei Sviridkin 7851969448 docs(gateway): document GatewayClass selection and edge TLS termination
An operator setting up a class whose provider terminates TLS upstream
had no page answering their questions: which platform values bound the
choice, what a tenant writes to make it, what a move between classes
does to certificates and to the replicated wildcard Secret, and how the
mechanism fails when a configured class name matches nothing installed.

Add docs/gateway-classes.md covering that surface. Describe the mode
and its consequences in the chart README alongside the other three,
including the one constraint stated nowhere else: the publishing tenant
cannot select a class whose TLS termination differs from the platform
default, because the cluster-wide ACME issuers derive their solver pin
from that default alone.

State the blast radius where it actually applies. The endpoints an edge
Gateway stops publishing exist only where those routes are rendered at
all, and both pages say what the transition does to the route
conditions the controller owns, so a reader is not left inferring it
from the code.

The security model names the selector each listener really carries. The
port-80 listener selects on the built-in kubernetes.io/metadata.name
label, which kube-apiserver writes and a namespace cannot forge; the
HTTPS and TLS-passthrough listeners select on
namespace.cozystack.io/gateway, written by the tenant chart and the
controller, so that guarantee does not extend to them. What closes the
exposure is that Namespace is cluster-scoped with no tenant verb on it,
and that the route-hostname policies bound what a labelled namespace
could publish.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:33:29 +03:00
Aleksei Sviridkin 57c51f58ee feat(gateway): stop replicating the wildcard key into edge namespaces
The wildcard-secret controller replicates the operator's TLS Secret,
private key included, into every namespace that owns a termination
point, and a namespace owning a Gateway counted as one without further
question. A Gateway whose class ends TLS at its provider's edge
terminates nothing locally, so the replica has no consumer there.

Read certMode off the tenant's own TenantGateway rather than a
cluster-wide value, because the class is a per-tenant choice. Keep the
key whenever the answer is not a confident no: a namespace that also
owns an ingress controller still terminates locally, a namespace
holding more than one TenantGateway keeps it while any of them is not
at the edge, and an unreadable list or a namespace whose TenantGateway
has not appeared yet keeps it too, so a transient API error never
withdraws a key a Gateway may still be serving.

Watch TenantGateway, so the withdrawal follows the class change rather
than waiting for the periodic resync. The watched set is a value the
tests assert on, because the controller-runtime builder exposes nothing
about what it watched: a dropped watch would otherwise leave both
suites green while the withdrawal quietly degraded to that resync,
leaving the private key in a namespace that stopped terminating TLS.

Pruning the replica is not the same as leaving the namespace key-free.
The Certificate objects go, but the Secrets they issued stay, as they
do on any other mode transition.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:26:54 +03:00
Aleksei Sviridkin 2ef0e8d3a1 feat(gateway): add an edge cert mode for upstream-terminated classes
Some GatewayClasses have their provider terminate TLS in front of the
Gateway and reach it over plain HTTP. On such a class the platform
issued origin certificates nobody serves, and where ACME could not
complete, the Gateway's listeners never became ready.

Add certMode=edge to TenantGateway. The controller renders the apex,
its wildcard and every inheriting child apex as plain HTTP listeners
carrying the same label-based allowedRoutes as the HTTPS listeners of
the other modes, so application HTTPRoutes attach by hostname
unchanged. It mints no Issuer and no Certificate, and removes the
http-to-https redirect route, because the edge performs that redirect
itself. TLSPassthroughServices are not rendered: an HTTP-only edge
cannot carry a TLS listener, which leaves the endpoints relying on one
unpublished through such a Gateway.

The hostname-less port-80 listener is left out. It carries the ACME
challenge and the redirect in the other modes and has neither here, so
it would only be a catch-all admitting any hostname. Leaving it out
also keeps a hostname-pinned listener at index 0, which is what
Cilium's first-listener-wins namespace check reads before the fix in
cilium/cilium#45693.

Moving into the mode garbage-collects the Issuer and Certificate
objects the controller owned. The TLS Secrets they issued stay behind:
cert-manager runs with enableCertificateOwnerRef false, so a
Certificate does not own the Secret it issued into. Every mode
transition already behaves that way.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-09-04 14:26:54 +03:00
Aleksei Sviridkin a2cb47c961 feat(gateway): update the CRD bundle to Gateway API v1.6.1 and gate GRPCRoute hostnames (#3872)
## What this PR does

Bumps the vendored Gateway API CRD bundle in
`packages/system/gateway-api-crds` from v1.5.1 to v1.6.1. The package
Makefile pins the upstream ref, so the change is that pin plus the
re-rendered `templates/crds-experimental.yaml`.

The bundle version is not only bookkeeping: it is a compatibility signal
consumers read at runtime. Every CRD in the bundle carries a
`gateway.networking.k8s.io/bundle-version` annotation, and a Gateway API
controller can read it off the `gatewayclasses` CRD to decide whether
the installed bundle matches the one it was built against. Controllers
that do this typically accept any patch release but require an exact
major and minor match, and report `SupportedVersion=False` with reason
`UnsupportedVersion` on their GatewayClass when the minor differs, a
status condition that surfaces the skew rather than a functional break.
The Cloudflare Tunnel Gateway API controller proposed for the platform
in #3858 is built against Gateway API v1.6.1 and performs exactly that
comparison, so against the v1.5.1 bundle shipped today its GatewayClass
reports the bundle unsupported, and with this bump it reports supported.
Neither change depends on the other landing first: this PR only moves
the bundle, and the check lives entirely on the consumer side.

One consequence worth stating plainly, since it will outlive this PR: an
exact-minor check makes the shipped bundle version a coupled pin for any
such controller. A future bump to v1.7.x flips the same condition back
to `UnsupportedVersion` until that controller is rebuilt against the
newer bundle, so the two want to move together.

The bundle still emits the same twelve CustomResourceDefinitions and the
same two `safe-upgrades.gateway.networking.k8s.io` admission objects,
and every CRD still carries the `helm.sh/resource-policy: keep` stamp
the Makefile's post-processing step adds. One of the policy's two CEL
rules is rewritten, though, and it changes what a cluster accepts. That
has its own section below.

Two kinds change their version surface: TCPRoute and UDPRoute gain a
served `v1` that takes over as the storage version, and their `v1alpha2`
stays served with a deprecation marker. Every other kind keeps exactly
the versions it served at v1.5.1. That matters because the Cilium
operator resolves GatewayClass, Gateway, HTTPRoute and GRPCRoute in
`v1`, ReferenceGrant in `v1beta1` and TLSRoute in `v1alpha2` before it
enables its Gateway API controller; all six are still served by the
v1.6.1 bundle. TLSRoute in particular keeps `v1alpha2`, which the tenant
gateway controller, the platform TLSRoute templates and the tenant
hostname admission policy all address.

Schema changes are small and land on fields nothing in this repository
sets. `sessionPersistence.idleTimeout` is gone from HTTPRoute, GRPCRoute
and XBackendTrafficPolicy, removed upstream under GEP-1619 because no
implementation ever implemented it and HTTP cookies have no native
idle-timeout mechanism, so no working behaviour is lost even though the
field disappears from the schema; HTTPRoute gains a lower bound on
`retry.attempts` and now rejects duplicate `retry.codes`; ReferenceGrant
requires `spec` at the root. In the other direction, Gateway
infrastructure annotations and frontend CA certificate references go
from a limit of 8 to 16, and TLSRoute hostnames from 16 to 1024. No CRD
schema loses or changes a CEL validation rule, and the six added belong
exclusively to the new TCPRoute and UDPRoute `v1` schemas.

### The safe-upgrades admission policy accepts a different set of
bundles

The experimental kustomization ships a `ValidatingAdmissionPolicy` and
its binding, both named `safe-upgrades.gateway.networking.k8s.io`, next
to the twelve CRDs. The binding is `validationActions: [Deny]` with
`failurePolicy: Fail`, matching CREATE and UPDATE on
`apiextensions.k8s.io/v1 customresourcedefinitions` cluster-wide, and
Cozystack installs it in two places: on the host through
`cozystack.gateway-api-crds`, and inside every guest cluster that sets
`addons.gatewayAPI.enabled`.

Its second rule reads the `gateway.networking.k8s.io/bundle-version`
annotation off an incoming CRD. At v1.5.1 that rule was a blacklist:
reject anything matching `v1.[0-4].\d+` or containing `v0`, allow the
rest. At v1.6.1 it is a whitelist: allow `v0.0.0-dev` outright,
otherwise require the annotation to start with `v1.` and not match
`^v1\.[0-4](\.|$)`. Evaluating both expressions over a spread of
annotation values:

| bundle-version | v1.5.1 policy | v1.6.1 policy |
| --- | --- | --- |
| `v1.6.1`, `v1.5.1`, `v1.5.0-dev`, `v1.10.0` | allow | allow |
| `v1.4.0`, `v1.4.10`, `v1.0.0` | reject | reject |
| `v0.0.0-dev` | reject | **allow** |
| `v2.0.0` | allow | **reject** |
| `1.6.1` (no leading `v`) | allow | **reject** |
| `v1.4` (no patch) | allow | **reject** |
| `main`, empty string | allow | **reject** |

Nothing in this repository trips the new rejections: every CRD the
bundle ships is stamped `v1.6.1`. What is exposed is an operator
applying a Gateway API CRD set from somewhere else on a cluster that
carries the policy. A fork, a vendor build or a nightly whose annotation
reads `1.6.1` or `v2.0.0` used to go in and is now denied with `reason:
Invalid`, naming a policy the operator may not know is installed, and
whatever HelmRelease owns those CRDs then sits un-Ready. Uninstalling
the policy is the escape hatch, and the denial message says so.

The same hunk fixes an upstream inconsistency: at v1.5.1 the policy pair
was stamped `v1.5.0-dev` while its twelve CRDs said `v1.5.1`. At v1.6.1
all fourteen documents agree.

The Gateway and HTTPRoute objects the tenant gateway controller renders
were serialized across all three certificate modes and checked against
both the v1.5.1 and the v1.6.1 schemas; the two runs are identical, so
the bump does not change what the controller can create. `go.mod` still
pins `sigs.k8s.io/gateway-api` v1.4.1 and this PR deliberately leaves it
there. The CRD bundle and the Go client types have been on separate
versions since before this change, and nothing the controller emits
depends on the gap.

One thing to know before the next bump: from v1.6 on, the two things
upstream calls the experimental bundle disagree. This package follows
`config/crd/experimental`, the kustomization, which emits 12 CRDs at
v1.6.1; the published `experimental-install.yaml` for the same tag emits
13, the extra one being `xbackends.gateway.networking.x-k8s.io`, new in
v1.6 and not listed in the kustomization. At v1.5.1 the two agreed, so
there was nothing to choose between. Nothing in this repository
references XBackend and no vendored CRD depends on it, so following the
kustomization costs nothing today, but whoever bumps next inherits the
choice, and it should be made deliberately rather than by whichever
command gets typed.

The second commit corrects a comment this bump invalidated. It claimed
TLSRoute stays at `v1alpha2` through Gateway API v1.5 and that the
promotion to `v1` was still upstream-only; TLSRoute has been served at
`v1` since v1.5.0, so the claim was already wrong before this bump and
the version range it named is now wrong too.

The third commit closes a gap that correction exposed.
`cozystack-route-hostname-policy-tls` named `v1alpha2` alone while the
bundle serves `tlsroutes` at `v1`, `v1alpha2` and `v1alpha3`: three
served versions against one named. The gap is not new and this bump
neither creates nor widens it: the v1.5.1 bundle and the v1.6.1 one
serve the same three versions, with the same two marked deprecated.
Coverage of the two unnamed versions rested entirely on request
conversion, because `matchPolicy` is unset on both rules in that file
and so defaults to `Equivalent`, which rewrites a request submitted
under any served version into the one the rule names before the policy
sees it. That is why there is no bypass to demonstrate, and equally why
the arrangement is brittle: the conversion has nothing left to convert
into once no named version is served, and a rule that selects nothing is
not caught by `failurePolicy: Fail`, because nothing failed. The rule
now names all three, which is the shape the HTTPRoute rule above it
already used, and `hack/route-hostname-policy-version-coverage.bats`
reads the served versions out of the vendored bundle and fails when a
policy does not cover every one of them, so a future bundle move cannot
outrun the list silently, in either direction.

That guard carries negative fixtures rather than only watching a correct
tree pass. Three of them drive `assert_covers` itself, with the two
readers redefined in a subshell, against an omitted served version, an
empty served set and an empty named set. Each requires the specific
complaint text rather than merely a non-zero exit, so a helper renamed
out from under it fails for the right reason instead of reading as a
successful rejection. A fourth drives the comparison helper directly.

The fourth commit pins the vendored bundle version in the CRD package's
unit suite. Nothing outside the package Makefile named a Gateway API
version, so re-running `make update` against a different ref landed
green: twelve CRDs, every one stamped, document count unchanged, suite
passing. The new case reads the
`gateway.networking.k8s.io/bundle-version` annotation off every rendered
CRD and fails when it is not the version the Makefile pins.

The fifth commit extends the version-coverage guard to
`cozystack-gateway-hostname-policy`, which names the `gateways` versions
literally the same way the two route policies do, and whose version set
lives in the vendored bundle and moves on `make update` there. It
matched the served set already; the guard now says so on every run
instead of leaving one of the three hostname policies unpinned.

Two notes for whoever reads the release: the TCPRoute and UDPRoute
storage version moves from `v1alpha2` to `v1`, and while nothing in this
repository ships either kind and the CRDs declare no conversion block,
an upgraded cluster accumulates both entries in `status.storedVersions`
— inert today, load-bearing only if a later bundle stops serving
`v1alpha2`.

The sixth commit gives GRPCRoute the same hostname rule the other two
route kinds have. GRPCRoute carries `spec.hostnames` and it can reach a
tenant Gateway: the port-443 listeners pin `allowedRoutes.kinds` to
HTTPRoute and TLSRoute, but the port-80 listener leaves `kinds` unset,
and an unset list resolves to the kinds the listener protocol supports,
which for HTTP includes GRPCRoute. A GRPCRoute in a tenant namespace
could claim any hostname on the plaintext listener with nothing checking
it. Tenants hold no `gateway.networking.k8s.io` RBAC in Cozystack, so
this is the same defence-in-depth against a chart bug that the other two
rules are, not a tenant-user control. The rule is the HTTPRoute rule
with `grpcroutes` in place of `httproutes` and the one version the
bundle serves for it, and the coverage guard pins that list to the
bundle like the others.

Several places in the tree enumerate which route kinds the hostname
policies cover, and all of them named two: the port-443 kinds invariant
in the tenant gateway controller and the test that pins it,
`packages/extra/gateway/README.md`, and the threat model together with
the self-assessment. They name three now. The port-443 kinds set also
carries an accurate reason now. cilium#45559 governs only that every
port-443 listener carries the same set, not what is in it, so the
contents are a least-privilege choice: nothing the platform ships needs
gRPC, TCP or UDP routing on port 443. The README had also described
HTTPS and TLS-passthrough listeners as carrying different kinds, where
the controller gives both the same one.

One more commit makes the suite's existing assertions bite. A
ValidatingAdmissionPolicyBinding whose `policyName` points at another
policy renders without complaint and leaves its own policy inert, and
the suite pinned `validationActions` on the bindings while never pinning
`policyName` at all, so pointing the GRPCRoute binding at the HTTPRoute
policy kept every case green with the GRPCRoute control doing nothing.
All three bindings now pin the policy they name. In the same file,
`uncovered()` assigned to the variable names its caller reads back when
printing a mismatch, so a call site with the arguments swapped would
have printed a diagnostic naming the wrong side.

Two things are named rather than made. The version-coverage guard
collects `apiVersions` from every `resourceRules` entry mentioning the
resource without also matching on `apiGroups` or `operations`, so a rule
in another group, or one scoped to a single verb, would donate its
versions to the set coverage is measured against; the chart's own suite
pins the group and the resource on the rule it checks, which closes that
a layer below. And
`packages/system/cozystack-api/templates/api-tlsroute.yaml` renders
`gateway.networking.k8s.io/v1alpha2` while the bundle serves `v1` as
storage and marks `v1alpha2` deprecated, but the identical pin sits in
the kubevirt and kubevirt-cdi TLSRoute templates, and the pinned
`sigs.k8s.io/gateway-api` carries no `apis/v1` TLSRoute type for the
controller side to follow, so moving one of the three here would leave
the tree less consistent than leaving all three.

Closes #3789.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, file by file: neither `packages/system/gateway-api-crds/Makefile`
nor its rendered CRD template matches any entry. Leaving every box empty
rather than ticking the "no downstream repository" line, because the
docs site carries a Gateway API version statement that is already
inaccurate and needs a decision that is not mine:
`content/en/docs/next/networking/gateway-api.md` and its v1.5 and v1.6
copies say Gateway API v1.5 ships TLSRoute at `v1alpha2` and that the
graduation to `v1` is still upstream-only, when v1.5.0 already shipped
TLSRoute `v1` as the storage version.

### Release note

```release-note
chore(gateway-api-crds): update the vendored Gateway API CRD bundle to v1.6.1. TCPRoute and UDPRoute gain a served v1 that becomes their storage version, with v1alpha2 still served and now deprecated; every other kind keeps the versions it served at v1.5.1. Upstream removed `sessionPersistence.idleTimeout` from HTTPRoute, GRPCRoute and XBackendTrafficPolicy; the field is not set by anything Cozystack renders, but a hand-written route that carries it loses it silently on its next write, with no admission error.

The bundle's `safe-upgrades.gateway.networking.k8s.io` ValidatingAdmissionPolicy changes which CRDs it accepts. Cozystack installs it cluster-wide on the host and in every guest cluster with `addons.gatewayAPI.enabled`. Its `gateway.networking.k8s.io/bundle-version` rule was a blacklist on two patterns and is now a whitelist on the `v1.` prefix with a `v0.0.0-dev` escape hatch. A bundle annotated `v0.0.0-dev` is now accepted where it was denied. A bundle annotated `v2.0.0`, `1.6.1` without the leading `v`, `main`, `v1.4` without a patch, or carrying an empty annotation is now denied where it was accepted. Nothing Cozystack ships is affected, since every CRD in the bundle is stamped `v1.6.1`, but applying a Gateway API CRD set from a fork, a vendor build or a nightly on such a cluster can now fail admission with `reason: Invalid`. Uninstalling the policy remains the documented escape hatch.

fix(cozystack-basics): the tenant TLSRoute hostname admission policy now names every served TLSRoute version (`v1`, `v1alpha2`, `v1alpha3`) instead of `v1alpha2` alone. Coverage of the other two rested on request conversion, which holds only while a named version stays served, and a rule left selecting nothing is not caught by `failurePolicy: Fail`.

feat(cozystack-basics): GRPCRoute hostnames in `tenant-*` namespaces are now gated by the same admission policy that gates HTTPRoute and TLSRoute. A tenant Gateway's port-80 listener leaves `allowedRoutes.kinds` unset, which admits GRPCRoute by protocol default, so a GRPCRoute could claim any hostname on the plaintext listener unchecked. Tenants hold no `gateway.networking.k8s.io` RBAC, so this is defence-in-depth against a chart bug rather than a tenant-user control.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
  - Added hostname admission validation for `GRPCRoute` resources.
- Expanded `TLSRoute` hostname validation to support all served Gateway
API versions.
- Ensured route hostname policies consistently cover HTTP, TLS, and gRPC
routes.

- **Security**
- Updated listener policies to restrict port-443 routes to supported
HTTP and TLS route types.
- Clarified fail-closed hostname validation behavior across Gateway,
Route, and Ingress resources.

- **Documentation**
- Updated security and threat-model documentation to include GRPCRoute
handling and validation coverage.

- **Maintenance**
  - Updated the bundled Gateway API CRDs to version 1.6.1.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-04 15:23:36 +04:00
Aleksei Sviridkin fb740db0b0 docs(security): separate the fail-open Gateway-listener policy from the fail-closed route policies
The self-assessment summarised the whole hostname policy set as
fail-closed, while its own table row states that the Gateway-listener
policy passes a Gateway whose namespace carries no host label. Say
which policies are which. The port-443 kinds comment in the reconciler
test also tied the GRPCRoute admission rule to the port-80 listener;
the rule matches the kind wherever it attaches, and the port-443
exclusion is a least-privilege choice, not the reason the rule exists.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-04 01:18:07 +03:00
Aleksei Sviridkin af19aa933c docs(security): record GRPCRoute in the route hostname policy set
The port-443 kinds set still excludes GRPCRoute, TCPRoute and UDPRoute.
cilium#45559 governs only that every port-443 listener carries the same
set, not what is in it, so the contents are a least-privilege choice:
nothing the platform ships needs gRPC, TCP or UDP routing on port 443.
TCPRoute and UDPRoute additionally carry no hostname and no admission
rule gates them.

The gateway README described HTTPS and TLS-passthrough listeners as
carrying different kinds. They carry the same one, which is what
cilium#45559 requires.

The threat model and the self-assessment enumerate which route kinds the
hostname policies cover, and both listed two.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 22:17:29 +03:00
Aleksei Sviridkin 9ea4a961ec fix(cacert): make the comments match the reconciler
Comments in the CA-extraction package described guards the code does
not have.

ManagedLabel: the anti-clobber decision is made by isOurProjection,
from the Secret type and an owner reference back to the sentinel, not
from the marker, which no decision keys on beyond the drift check.

managedByCozystackLabel: the reconciler does mutate it. It deletes the
label on a selectors-digest change, which is what hands the projection
back to the lineage webhook for a fresh verdict.

projectionSuffix: the projection name is kept clear of engine CA
Secrets by a dot, and a release name cannot carry one, but the suffix
an operator appends can and nothing validates it. A suffix ending in
".tenant-ca" is the only one that reaches a canonical name.

TestProjectionNameCannotCollideWithADotFreeEngineSuffix: both its doc
and its name claimed the property for any engine suffix while the loop
samples only dot-free ones, which is the same overstatement one file
over.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-09-03 21:35:01 +03:00
Andrey Kolkov e94fcf8219 feat(backups): RabbitMQ definitions backup via a dedicated strategy driver (#3990)
## What this PR does

Adds backup/restore support for the managed `RabbitMQ` application.
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so this backs up broker
**definitions** (vhosts, users, permissions, queues, exchanges,
bindings, policies, parameters) over the management HTTP API
(`GET`/`POST /api/definitions`) and stores them in the system bucket.
Message payloads and point-in-time recovery are intentionally out of
scope; a full-data backup would be a Velero volume snapshot, which is a
separate strategy.

It is delivered as a **first-class
`strategy.backups.cozystack.io/Rabbitmq` driver**, following the
Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

- `reconcileRabbitmq` / `reconcileRabbitmqRestore` render the strategy's
`PodTemplateSpec` — `curl` `GET`/`POST /api/definitions`,
pushing/pulling the JSON to S3 with `curl --aws-sigv4` — as a
`batch/v1.Job`, covering both restore flows: in-place (into the source
broker) and to-copy (into a separate, differently-named RabbitMQ in the
same namespace, authenticating with the target's own default-user
credentials and reading the source's object key).
- A **Go-read precondition** gates each run on the `RabbitmqCluster`
reporting `AllReplicasReady` (read through a small `rabbitmqtypes`
partial type), so a not-yet-ready broker surfaces a precise
`Ready=False` wait on the BackupJob/RestoreJob rather than a Job that
starts and fails against an unreachable management API.
- The produced `Backup` records the stored object's location on
`status.artifact.uri`, rendered from the strategy's
`spec.artifactURITemplate`. The `Backup` CRD has no status subresource,
so this persists on Create, mirroring the Etcd driver.

Dispatch is wired in both controllers (plus
`supportedBackupStrategyKinds`), `rabbitmqtypes` is registered in the
scheme, and RBAC grants read on `rabbitmqclusters.rabbitmq.com` (the Job
itself is already covered by the existing `batch/jobs` grant). It ships
`cozy-default-rabbitmq` (a `$bucketName`-gated `Rabbitmq` strategy) and
routes `apps.cozystack.io/RabbitMQ` to it in the `cozy-default`
BackupClass, which also enrols it in the `DefaultObjectsGate`. The
RabbitMQ application chart is left untouched — it had no prior backup
wiring, and the strategy carries the S3 coordinates itself.

Unit tests cover the driver's pure helpers (applicationRef validation,
cluster-name derivation, the readiness-message builder, artifact-URI
rendering, parameter round-trip) and the reconcile precondition gate
(defers with `Ready=False` and no Job until the cluster is ready, then
proceeds). A runnable example under `examples/backups/rabbitmq/` and an
honest-roundtrip Chainsaw e2e under `hack/e2e-chainsaw/rabbitmq/` seed a
sentinel vhost, back it up, drop it, and assert it round-trips through
object storage on both restore flows.

The Job pod reuses the existing `platform-migrations` image (curl +
ca-certificates) under a new
`backupStrategyController.rabbitmqBackupClientImage` key, stamped in
lockstep with `chBackupClientImage` in `packages/core/platform/Makefile`
so the repository stays pinned at a single digest
(`hack/image-pin-consistency.bats`). A dedicated slim image built by
this package is a follow-up.

**Validation:** `go build`/`go vet`/`gofmt`/`go test
./internal/backupcontroller/... ./internal/template/...`, deepcopy +
Rabbitmq CRD generation, `helm template` proving the CRD-ordering gate
both on and off, `helm lint`, and `helm unittest` (18/18). The backup +
in-place + to-copy roundtrip was verified end-to-end on a dev cluster (a
real management-API export and a real S3 `--aws-sigv4` upload/download
against SeaweedFS), so the driver's core contract is exercised on a real
cluster; the in-CI Chainsaw run is the remaining automated evidence. The
object-cleanup, terminating-namespace teardown and readiness-deadline
paths added during review are unit-tested (including `Reconcile`-level
finalizer retention and the terminating-namespace release) but not yet
cluster-exercised. (Note found while testing: RabbitMQ 4.2 is OOMKilled
on the `nano` preset, so the example/e2e use `c1.small`.)

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map against the diff. This is an additive backup
strategy plus a new strategy CRD; no downstream repository is
mechanically forced by the change. A documentation page for RabbitMQ
backup on [cozystack/website](https://github.com/cozystack/website)
would be a reasonable follow-up, but it is not required by this diff —
leaving the boxes unticked for a human to decide rather than ticking "no
downstream" to clear the checklist.

### Release note

```release-note
feat(backups): add RabbitMQ definitions backup/restore via a dedicated strategy.backups.cozystack.io/Rabbitmq driver; it gates each run on the RabbitmqCluster being ready and records the stored object on Backup.status.artifact.uri.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added RabbitMQ backup and restore support for definitions, including
vhosts, users, queues, exchanges, bindings, policies, and related
settings.
  * Added in-place and restore-to-copy workflows with readiness checks.
  * Added scheduled and on-demand backup examples.
  * Integrated RabbitMQ backups with the default backup configuration.
* Backups store definitions only; message payloads and point-in-time
recovery are not included.
* **Documentation**
  * Added a complete RabbitMQ backup and restore demonstration guide.
* **Tests**
* Added end-to-end coverage for readiness and backup/restore round
trips.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-03 16:03:46 +04:00
Andrey KolkovandClaude Opus 4.8 ba32df69c0 feat(backups): add RabbitMQ definitions backup via a dedicated strategy driver
RabbitMQ's cluster operator ships no backup CRD and RabbitMQ has no
consistent online export of message payloads, so back up broker
DEFINITIONS (vhosts, users, permissions, queues, exchanges, bindings,
policies, parameters) over the management HTTP API and store them in the
system bucket. Message payloads and PITR are out of scope - a full-data
backup is a Velero volume snapshot, a separate strategy.

Add a first-class strategy.backups.cozystack.io/Rabbitmq driver, following
the Altinity pattern (a dedicated driver that renders a one-shot Job for an
HTTP-API engine):

  - reconcileRabbitmq / reconcileRabbitmqRestore render the strategy's
    PodTemplateSpec (curl GET/POST /api/definitions, push/pull to S3 with
    --aws-sigv4) as a batch/v1.Job, for both in-place and to-copy restore.
  - A Go-read precondition gates each run on the RabbitmqCluster reporting
    AllReplicasReady (via the rabbitmqtypes partial type), surfacing a
    precise Ready=False wait instead of a Job that starts and fails.
  - The produced Backup records the object location on status.artifact.uri
    (rendered from spec.artifactURITemplate; the Backup CRD has no status
    subresource, so it persists on Create, as in the Etcd driver).

Wire dispatch in both controllers + supportedBackupStrategyKinds, register
rabbitmqtypes in the scheme, and grant read on rabbitmqclusters. Ship
cozy-default-rabbitmq (a $bucketName-gated Rabbitmq strategy) and route
apps.cozystack.io/RabbitMQ to it in the cozy-default BackupClass. The
application chart is untouched.

Add unit tests for the driver's pure helpers and the precondition gate, a
runnable example, and an honest-roundtrip Chainsaw e2e that seeds a
sentinel vhost, backs it up, drops it, and asserts it round-trips through
object storage on both restore flows.

The Job pod reuses the platform-migrations image (curl + ca-certificates)
under a new rabbitmqBackupClientImage key, stamped in lockstep with
chBackupClientImage so the repository stays pinned at one digest; a
dedicated slim image is a follow-up.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-09-01 18:21:10 +04:00
Andrey KolkovandClaude Opus 4.8 5bcdb8cfb8 fix(postgres): let in-place restore re-bootstrap instead of wedging
An in-place RestoreJob (targetApplicationRef omitted) re-bootstraps the
Postgres app under its own name. On stock v1.6.2 it never comes back up,
for two independent reasons in the same restore patch - both surface only
in place, because a to-copy restore targets a different app.

1. Archive-vs-recovery serverName collision. The driver set the restored
   cluster's WAL-archive serverName equal to the recovery source
   (bootstrap.serverName = bootstrap.oldName = sourceServerName) while the
   chart pinned the archiver serverName to the release name. In place those
   coincide, so the cluster archived onto the source's own WAL prefix and
   barman-cloud-check-wal-archive refused to start ("Expected empty
   archive"). Fixed by giving the restored cluster a fresh
   bootstrap.newServerName, keyed by the RestoreJob UID (unique per
   restore, stable across reconciles); the chart archives there while
   externalClusters still recovers from bootstrap.serverName. This is what
   the manual CNPG strategies hit (serverName defaults to the cluster
   name); the cozy-default flow dodged it (serverName is <ns>-<app>).

2. useSystemBucket guard. The restore populates explicit S3 coordinates
   (destinationPath / s3CredentialsSecret / endpointCA) but left
   backup.useSystemBucket=true, so the chart's "bootstrap.enabled is
   incompatible with useSystemBucket" guard failed the HelmRelease upgrade
   and the cluster never re-rendered. This is what the cozy-default
   (system-bucket) flow hits on an in-place restore. Fixed by clearing
   useSystemBucket in the restore patch, since the app is now on the
   explicit-creds path. The driver's app view (postgresapp.Backup) gains
   the field, without omitempty so a false survives the merge patch.

Both are exercised only by an in-place restore; the postgres backup e2e
drove only to-copy, so neither surfaced in CI. A step 50 in-place restore
now guards reason 1 (manual-strategy flow); reason 2 is covered by a Go
unit assertion (a system-bucket in-place e2e needs the platform bucket in
CI and is left as a follow-up).

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-25 17:31:53 +04:00
Aleksei Sviridkin 3d076099a2 docs(postgres): name the trust anchor as the tenant actually sees it
The README told a tenant to fetch <release>.tenant-ca, but the release
name carries the postgres- prefix the application definition applies, so
the object is postgres-<name>.tenant-ca. Following the old text returned
NotFound.

Also drop two comments promising a companion ValidatingAdmissionPolicy
that pins sentinel writes to helm-controller. No such policy ships: the
proposal for it was rejected because authorization by username duplicates
RBAC in a more brittle form. The RBAC boundary is the whole control.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-08-17 03:10:24 +03:00
Aleksei Sviridkin 22ce998890 fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy (#3546)
## What this PR does

Fixes #3544.

`flux-shard-operator` clones the flux-aio `helm-controller` container
into each `helm-controller-shard<i>` Deployment. It already sanitises
the all-in-one localhost wiring, but keeps two things that stop the
standalone shard from starting behind a corporate HTTP proxy:

1. **Inherited `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`.** A standalone
shard needs no external egress (source-controller fetches artifacts).
Behind an unreachable proxy the controller makes a blocking startup
HTTPS call that never completes, the manager never serves `/healthz`,
and the liveness probe crashloops the pod forever. flux-aio survives
only because it starts once and never restarts. On an affected cluster
every HelmRelease carrying a `sharding.fluxcd.io/key` (150 of ~240) was
frozen for days, silently.

2. **No `startupProbe`.** The clone inherits only a ~30s liveness
window, so any slow start (large cache sync) is fatal and unrecoverable.

This PR extends the sanitisation to drop the proxy env, and derives a
`startupProbe` from the liveness handler with a generous budget so a
slow-but-progressing start is not killed. Verified by an A/B on a live
cluster (throwaway helm-controller with a non-matching selector): with
the proxy env it hangs; without it, it starts cleanly (`Starting
workers`).

### Downstream repositories
- [x] No downstream repository is affected by this change

Internal operator behavior only; no API/CRD/values contract change.

### Release note
```release-note
fix(flux-shard-operator): sharded helm-controller no longer crashloops at startup behind an HTTP proxy; the cloned controller drops the inherited proxy env and gains a startupProbe, so HelmReleases assigned to shards keep reconciling in proxied environments.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Prevented inherited proxy settings, including uppercase and lowercase
variants of `NO_PROXY`, from affecting Flux shard deployments.
* Automatically adds a startup health check when only a liveness check
is configured.
* Preserved the liveness check’s handler and timeout while applying
appropriate startup timing and failure settings.

* **Documentation**
  * Documented startup health checks for slow helm-controller startup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-14 13:28:51 +03:00
Andrey Kolkov b30c7f4272 fix(backups): requeue MongoDB restore on a transient source-CR read error
resolveMongoDBBackupSource folded two failure classes into one error return:
a transient API read error on the live PerconaServerMongoDBBackup CR (cache
not yet synced, apiserver hiccup) and the terminal no-restorable-destination
case. The RestoreJob caller turned every error into markRestoreJobFailed, so a
retryable read error failed a restore that would have succeeded on the next
reconcile. The backup path already treats the symmetric live-CR read error as
transient (return ctrl.Result{}, err).

Wrap the transient read error in a new errTransientRestoreSource sentinel and
requeue on it at the caller; only a genuinely unresolvable source stays
terminal. Adds a non-vacuous interceptor-driven test.

Addresses the MINOR robustness note from PR review.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey Kolkov b97c443a16 fix(backups): harden MongoDB restore auth + PITR option handling from review
Address the two blocking findings in PR #3562:

- Restore into a differently-named target now authenticates with the
  TARGET cluster's own S3 credentials instead of the source's. The
  backupSource carried the source release's -s3-creds Secret, which is
  deleted together with the source app — the disaster-recovery case
  backups exist for — leaving a to-copy restore unable to reach S3. The
  restore replays into the target (gated on backup.enabled just above),
  whose own -s3-creds Secret is present and grants access to the same
  bucket; only the credential reference is swapped, bucket/prefix/endpoint
  still come from the source archive. Falls back to the source reference
  when the target declares no discoverable s3 credentials (in-place
  restore resolves to the same Secret, so it is a no-op there).

- A non-string recoveryTime (spec.options is free-form runtime.RawExtension,
  so {"recoveryTime": 20260805} passes admission) no longer silently
  degrades a PITR request to a plain snapshot restore reported Succeeded.
  When the typed decode fails and the blob carries a recoveryTime key, the
  RestoreJob now fails terminally — the same guarantee pitrSpec already
  makes for a malformed RFC3339 string.

Minor findings also addressed:

- BackupJob tolerates a not-yet-present MongoDB app CR with the same
  bounded grace the psmdb-CR existence gate uses, instead of failing
  terminally on a GitOps single-commit ordering race.
- examples: the to-copy target writes into its OWN S3 prefix
  (mongodb-target/) rather than the source's, so the two clusters' oplog
  and backup metadata no longer interleave under shared keys (both run
  replset rs0).
- examples: wait_psmdb_ready no longer treats a transient state=error as
  terminal — the cluster state is recomputed each reconcile and can blip
  through error during replset init before converging to ready.

Unit tests cover the target-credential selector (preferred/default/sole
storage) and the load-bearing recoveryTime-key detection.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 0d4a3e3134 refactor(backups): make MongoDB PITR option uniform with CNPG + warn on unknown restore options
PITR: replace the bespoke spec.options.pitr{type,date} blob with a single
spec.options.recoveryTime RFC3339 string, matching the CNPG strategy's option
exactly (CNPGRestoreOptions.RecoveryTime). The driver converts recoveryTime to
the psmdb Restore's pitr{type:date,date} in the operator's "YYYY-MM-DD HH:MM:SS"
UTC format. An empty recoveryTime restores the snapshot as-taken (documented
difference from CNPG, whose empty value means latest WAL — a psmdb logical
backup is already consistent). A non-RFC3339 recoveryTime fails terminally.

Unknown options: parseMongoDBRestoreOptions now also returns the set of
spec.options keys the driver does not recognise; reconcileMongoDBRestore emits a
UnknownRestoreOption Warning event per unknown key (and logs it) instead of
silently ignoring it — so a "recoverytime" typo no longer silently degrades a
PITR restore to a snapshot restore. (BackupJob carries no spec.options blob, so
there is no backup-path equivalent; the only backup-time input is the free-form
BackupClass parameters used as template vars.)

Docs + examples updated to recoveryTime; tests cover the RFC3339→UTC mapping,
empty/invalid recoveryTime, and unknown-key reporting.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 540bb94f58 test(backups): cover createMongoDBBackupArtifact producer seam
Address branch-review blocker: add TestCreateMongoDBBackupArtifact_ArtifactShape,
mirroring the MariaDB sibling. createMongoDBBackupArtifact writes the
driver-metadata keys (backup-name/namespace/destination), the Status.Artifact
URI, and the persisted snapshot that the restore path reads back — the
producer half of a seam previously exercised only end-to-end. A key-name drift
between this producer and resolveMongoDBBackupSource would otherwise pass every
unit test and surface only in e2e. The test asserts the metadata keys, phase,
artifact URI, and that the snapshot decodes to a usable by-reference
backupSource; a second case pins the empty-destination path (nil Artifact, no
destination key).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 bc5e47c64f fix(backups): add explicit MongoDB cleanup-on-delete branch + its guard test
Address branch-review recommendation: give the MongoDB strategy an explicit
case in BackupReconciler.cleanupOnDelete, matching CNPG / Altinity / MariaDB /
FoundationDB. A deleted MongoDB Backup artifact previously fell through to the
Velero default (a safe no-op — MongoDB driverMetadata never carries
velero.io/backup-name), but the explicit branch documents the "we do not own
the psmdb S3 archive" contract and guards the seam against a future shared
helper incidentally stamping velero.io/backup-name onto MongoDB metadata.

Ships with TestBackupCleanup_MongoDB_DoesNotFallThroughToVelero, mirroring the
one-test-per-branch convention its three siblings follow: it seeds the velero
metadata keys + a matching velero.io/Backup and asserts no DeleteBackupRequest
is synthesised and the Velero Backup survives — so deleting the branch makes
the test fail, which is what makes the guard self-verifying.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Andrey KolkovandClaude Opus 4.8 dff56aa998 feat(backups): add a native MongoDB (psmdb) backup-strategy driver
Adds a MongoDB (Percona Server for MongoDB / psmdb) backup-strategy driver so
`apps.cozystack.io/MongoDB` reaches feature parity with the CNPG, MariaDB and
ClickHouse native drivers — logical backups, restore-to-differently-named
instances, and point-in-time recovery through the unified
BackupJob/RestoreJob/BackupClass/Plan interface (closes #3545).

The driver orchestrates the operator's PerconaServerMongoDBBackup /
PerconaServerMongoDBRestore CRDs:

- BackupJob: creates a one-shot PerconaServerMongoDBBackup, polls
  status.state, and persists the S3 destination + storage descriptor
  (by-reference, no credentials) onto the Cozystack Backup artifact.
- RestoreJob: drives PerconaServerMongoDBRestore via backupSource (read back
  from the source backup status, snapshot fallback) so the same artifact
  restores in-place or into a differently-named instance; PITR via
  spec.options.pitr (date/latest).

Because psmdb resolves the backup storage by name against the cluster's
spec.backup.storages, the driver names the chart's storage (s3-storage) via the
cozy-default-mongodb Strategy rather than injecting it, and surfaces a clear
Ready=False precondition when a target has backups disabled.

Wiring:
- api/backups/strategy/v1alpha1: MongoDB strategy type + generated CRD/deepcopy.
- internal/backupcontroller: psmdbstrategy_controller + psmdbtypes/mongodbapp
  partial-schema mirrors; dispatch, supported-kinds and scheme registration.
- backupstrategy-controller: cozy-default MongoDB route, cozy-default-mongodb
  Strategy CR, and psmdb.percona.com RBAC.
- apps/mongodb: backup.insecureSkipTLSVerify knob (psmdb s3 storage has no
  CA-bundle field) so backups can target Cozystack's self-signed seaweedfs.

Tests + e2e:
- unit tests for the driver (validation, ensure-idempotency, precondition,
  backupSource resolution, snapshot round-trip, PITR option parsing).
- examples/backups/mongodb + a mongodb-2-backup-roundtrip Chainsaw suite that
  drives run-all.sh end-to-end (sentinel document round-trips through S3 into a
  restored copy while the source stays untouched), per docs/agents/e2e-testing.md.
- docs/operations/backup-classes.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-12 12:00:33 +04:00
Aleksei Sviridkin ec6a835251 fix(manifestutil): report one message when the CRD wait is cancelled (#3742)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

`WaitForCRDsEstablished` reported a cancelled context two different
ways. The check at the top of the poll loop returned a message with no
CRD name, the `select` at the bottom named the CRD the poll had stopped
on. When a tick and the deadline come ready in the same instant,
`select` picks between the ready cases at random, so one cancellation
reached the operator either way.

Both callers wrap this as `CRDs not established after apply`, so the CRD
name is what makes that log line actionable, and whether it is there was
a coin toss. `TestWaitForCRDsEstablished_timeout` asserts on the name,
which is where this surfaced: its 2s deadline is an exact multiple of
the 500ms ticker, so the two channels come ready together on the fourth
tick and the assertion failed at random.

The message is now built in one place and returned from both exits, so
the two are indistinguishable by construction rather than by inspection.
The name is carried across iterations, because a cancellation noticed at
the top of the loop has no poll of its own to take one from, and it
starts as the first CRD of the set for a wait cancelled before any poll
observed anything. That start value is a stand-in rather than an
observation, and the comment on it says so; the alternative, an empty
name and a second phrasing for it, would leave the loop-top message with
nothing to assert against and no test able to catch its return to the
unnamed form.

Deleting the top check instead, so there is literally one exit, is the
smaller diff and it moves the name to the head of the list in the very
interleaving this PR is about. A client refuses `Get` on a dead context
at the first name in the list whichever CRD is actually lagging, so a
poll entered on a cancelled context renames the report to that first
name. That premise is about the production client, `client.New` at
`cmd/cozystack-operator/main.go:176`, uncached, so the request is
rejected before it is issued, and it is worth saying that nothing in the
test surface pins it: replacing the `ctx` passed to `Get` with
`context.Background()` survives the whole package, and the fixtures that
need a context-honouring client check `ctx.Err()` by hand. That line
predates this change and is untouched by it. Measured with a client that
honours the context: with the check, a wait that re-enters the loop
after the deadline keeps the name its last poll settled on; without it,
the report moves to the head of the list. The check closes that window
and no more than that. A context that dies part-way through a poll still
records the first `Get` to fail, exactly as merge base does, because
this branch cannot tell a refused request from a CRD status. That is
unchanged here and written up separately.

Four tests cover the ways this wait can end on a cancelled context:
before any poll has observed the set, cut off inside a poll, cancelled
while waiting for the next tick, and re-entering the loop on a context
that died during that wait. Two of them are red on merge base and are
the regression pins for the defect above, both deterministically rather
than by repetition: merge base answers a wait that starts cancelled, and
a wait that re-enters the loop cancelled, with the unnamed message every
time. Its loop-top exit is a `select` whose `Done` case is ready, and a
ready case always beats `default`. The middle two are green on merge
base, which gets those cases right; they are here because each catches a
specific way of breaking the fix. Every one of the four is the sole
killer of at least one mutation, which is the property worth having: it
says none of them is redundant. They compare the whole message rather
than searching it for a name, because a second exit that happens to
carry a name is still the defect being fixed here, and a test that only
looks for the name cannot see it.

The re-entry test needs the bottom `select` to take the tick while
cancellation is already pending, which against a real context is the
same coin toss the bug rides on. The double it uses hands out an open
`Done` for the single read that `select` makes in that iteration and a
closed one for every read after, so the tick wins by construction and
the next iteration sees a context cancelled through both `Done` and
`Err`, the way a real one is. That pairing is guaranteed rather than
incidental: `cancelCtx.Err` reads `<-c.Done()` before returning a
non-nil error, so a double that reported cancellation through `Err`
alone would be modelling a state no context can be in, and would pin the
shape of the check rather than its behaviour. One property of that
double is worth knowing before editing the loop: it hands out its open
and closed channels per read, so a `Done()` read added inside the poll
would consume the open one and leave the bottom `select` a closed
channel, quietly turning this into a second bottom-exit test that still
passes. The failure direction there is false-green, which is the worse
half, and the cheap guard is a lower bound on elapsed time inside that
test, a run that leaves through the loop top spends a ticker interval
getting there, one that leaves at the bottom of the first iteration does
not. Not done here, because the test is correct today and provably kills
the mutation it exists for, so the hazard is conditional on a future
edit.

The timeout test does reproduce locally under load: 6 failures across
400 runs on the unfixed code while I was measuring, spread over five
batches, the last of which saw none. #3719 reports that it does not
reproduce locally, so this is worth writing down before someone spends
an evening re-testing that. That spread is also why the number is an
order of magnitude and not a rate.

`TestInstall_crdNotEstablished` in `internal/crdinstall` has the same 2s
deadline against the same 500ms ticker. It asserts on the `CRDs not
established` wrapper the caller adds, and both messages carried that
wrapper, so this coin flip could not reach it and nothing here touches
it.

I left the 2s deadline where it is. Moving it off the tick boundary
would have hidden the coin flip and left the second message in place.
The new tests catch a reintroduced second message deterministically,
which is what that job now rests on rather than the timeout test's
roughly one-in-sixty.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(manifestutil): a CRD wait cut short by a cancelled context now reports one error naming the CRD it was waiting on, where before the name was present or missing depending on which of two exits the scheduler happened to pick
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved cancellation handling while waiting for custom resources to
become established.
* Cancelled operations now stop before issuing unnecessary requests and
identify the affected resource.
  * Timeout errors remain distinguishable from cancellation errors.

* **Tests**
* Added coverage for cancellation before, during, after, and between
polling attempts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-11 02:53:40 +03:00
Aleksei Sviridkin 5747747087 fix(manifestutil): report one message when the CRD wait is cancelled
WaitForCRDsEstablished reported a cancelled context two different ways.
The check at the top of the poll loop returned a message with no CRD
name; the select at the bottom named the CRD the poll had stopped on.
Which one ran was left to the scheduler: when the deadline and a tick
come ready in the same instant, a select picks between the ready cases
at random, so one cancellation reached the operator either way.

Build the message in one place and return it from both exits, so the two
are indistinguishable by construction. The name is carried across
iterations, since a cancellation noticed at the top of the loop has no
poll of its own to take one from, and starts as the first CRD of the set
for a wait cancelled before any poll observed anything.

Keeping the check at the top is what makes carrying the name worthwhile.
A client refuses Get on a dead context at the first name in the list
whichever CRD is actually lagging, so a loop that re-entered and polled
anyway would overwrite the name its last poll settled on. The check
closes that window and no more than that: a context dying part-way
through a poll still records the first Get to fail, because this branch
cannot tell a refused request from a CRD status.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
2026-08-10 15:36:05 +03:00
IvanHunters 66df3b20e0 docs(flux-shard-operator): drop unproven mechanism from proxy comments
Name all six proxy env spellings in the sanitisation list, and replace
the unestablished 'external HTTPS call' / 'flux-aio does not restart'
claims with the grounded in-cluster mechanism: after the
KUBERNETES_SERVICE_HOST drop above, the shard talks to the management
apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does
not match, so that startup call is what stalls behind an unreachable
proxy. Comment-only; no behaviour change.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-10 12:56:42 +03:00
IvanHunters a10fcb3e4a test(flux-shard-operator): assert full startupProbe contract and reject alias regression
Model the source liveness probe as flux-aio actually ships it (a bare httpGet
with no timing fields) and tighten the startupProbe assertions:

- assert the complete normalised budget (delay/period/success/failure) instead
  of only FailureThreshold, so a later PeriodSeconds regression cannot silently
  restore a short crashloop window;
- assert TimeoutSeconds is inherited from the source probe rather than a
  hardcoded value, so forcing it stricter is caught;
- assert the startupProbe is a DeepCopy of liveness, not an alias, which would
  otherwise stamp the 30-failure startup budget onto liveness and produce a
  never-failing liveness probe.

The alias assertion is non-vacuous: it fails when the DeepCopy is regressed to
a plain alias.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-07 16:33:21 +03:00
IvanHunters 036a31b550 docs(flux-shard-operator): record proxy-env drop and startupProbe in sanitisation list
The BuildShardDeployment doc comment and the operator README are the only
record of how far the cloned shard container has diverged from flux-aio.
Add the two steps this fix introduced (dropping the inherited corporate-proxy
env and deriving a startupProbe) so the list stays the authoritative account
and does not invite re-adding what was deliberately removed.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-07 16:33:13 +03:00
Mattia Eleuteri f859b710a4 fix(backupstrategy-controller): repair the bucket credentials Secret from the gate, not a chart fail
Addresses the four findings on #3524.

**The `fail` in the bucket chart (MAJOR).** The <bucket>-<user>-credentials
Secret was made to `fail` the render while its COSI lookup was empty, so the
release would retry and converge instead of skipping the Secret permanently.
Helm cannot render a partial set, so that fail aborts the WHOLE
<bucket>-system release: the other users' Secrets, and the bucket UI
Deployment/Service/Ingress/HTTPRoute with them. One declared user whose
BucketAccess never provisions takes down every other user of that bucket,
and on an already-installed release parks it in Failed, blocking every later
upgrade.

**No per-bucket escape (MAJOR).** Rather than add one, remove the need for
it. The gate already forces the release that renders the lookup-gated
Strategy CRs; the credentials Secret is the same trap one release earlier,
and the gate reads that Secret to resolve the bucket name in the first
place. So while it is absent, or carries no bucket name, force the bucket's
<bucket>-system release instead of doing nothing. That cannot deadlock its
own precondition: the BucketClaim and the BucketAccess whose COSI Secret the
lookup reads are rendered unconditionally by the parent release, not by the
one being forced.

The two releases are throttled independently — they are forced in sequence
on a bootstrap, so a shared timestamp would delay the second by a full
MinForceInterval for no reason. Coordinates come from the chart under
provisionBucket, and are omitted on external S3 where the Secret is
admin-managed and no release renders it. No RBAC delta: patch on
helmreleases is already cluster-scoped.

The chart goes back to a partial render, with the comment now explaining why
the skip is not self-healing and why failing is the wrong lever, and a test
that renders the UI with every user unresolvable so a reintroduced `fail`
fails the suite. Tenant buckets keep their pre-existing behaviour;
generalising the repair to every bucket needs a controller that owns them
and is tracked separately.

**Stale gauge (MINOR).** cozystack_backup_default_objects_missing was only
written on the happy path, so the state the gate exists to catch reported 0
and the documented alert never fired. It is now written on the unresolved
path too — the credentials Secret counts as one of the objects the default
backups depend on, which is what it always was. On an API error the gauge is
deliberately left alone rather than flapping, so a new
cozystack_backup_default_objects_check_errors_total marks it stale, and the
doc pairs the two instead of overstating the gauge.

**Suspended releases (MINOR).** forceHelmRelease now does a point Get and
skips a release with spec.suspend: true. helm-controller ignores forceAt and
requestedAt while suspended (`cozyhr suspend` sets exactly that), so the gate
was re-stamping every MinForceInterval for the whole suspension and climbing
the force counter — which the runbook attributes to a render that is not
producing the objects, the wrong diagnosis. The skip is logged and not
counted, so a climbing counter keeps its documented meaning.

Reported-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-By: Claude <noreply@anthropic.com>
2026-08-07 11:10:41 +02:00
IvanHunters 8b006391c2 test(backupstrategy-controller): mirror all six cozy-default routes in the gate fixture
The gate reads the BackupClass as the manifest of what must exist, but the Go
fixture listed only CNPG, Etcd and the two Velero routes while the shipped
backupclass-default.yaml also routes MariaDB and Altinity. Add both to the
fixture, and to the RESTMapper and list-kinds derived from it, so the
expected-object set the tests exercise equals the real route set.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 10:09:04 +03:00
IvanHunters bf152ba864 fix(backupstrategy-controller): resolve the default BSL through the RESTMapper
The BSL existence check used a hardcoded GVR and the dynamic client, whose
Get returns a plain 404 (IsNotFound) for an unserved API group. The
IsNoMatchError branch meant to skip an absent Velero API was therefore dead
code, and if Velero were uninstalled after bootstrap while bslEnabled=true
the BSL would be counted missing and force a Helm upgrade forever. Route the
lookup through the RESTMapper like the strategy loop, so an absent velero.io
API is a NoMatch we skip. A regression test fails without the change.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 09:55:12 +03:00
IvanHunters aa3accd128 fix(backupstrategy-controller): stop the default-objects gate looping when velero.bslEnabled=false
The cozy-default BackupClass routes VMInstance/VMDisk to the Velero
Strategy CRs unconditionally, but the chart gates those CRs on
velero.bslEnabled. With the BSL disabled they never render, so the gate
counted them as permanently missing and forced a Helm upgrade every
MinForceInterval forever, climbing force_reconciles_total and pinning the
missing gauge. Skip Velero-kind strategyRefs when the BSL is disabled
(empty VeleroNamespace), the same flag that gates their render. The
regression test now omits the manually-created Velero objects so it fails
without the skip. Correct the values.yaml comment that wrongly claimed the
Strategy CRs still ship in this mode.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 09:52:18 +03:00
IvanHunters affdd70e9d fix(flux-shard-operator): inherit probe timeout, correct comments
Review follow-ups:
- Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value
  inherited from the liveness handler, otherwise a future flux-aio with a
  larger liveness timeout would get a stricter startup probe and could
  recreate the crashloop. Covered by a regression guard (liveness timeout 5s
  must be inherited).
- Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking
  external HTTPS call at startup) and note the remote spec.kubeConfig
  exception that does not apply to cozystack in-cluster guest apiservers.
- Correct the test fixture comment: the proxy env is operator/patch-level, not
  installer-injected (the installer injects only KUBERNETES_SERVICE_*).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 23:22:59 +03:00
IvanHunters d44c87d939 fix(flux-shard-operator): add startupProbe to cloned helm-controller
The cloned shard Deployment inherits only a liveness probe (~30s window,
no startupProbe), so a controller still syncing caches on startup is killed
before it serves /healthz and never recovers. Derive a startupProbe from the
liveness handler with a generous budget so a slow start is not fatal, while
liveness still catches a wedged running pod.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 23:08:39 +03:00
IvanHunters c475faae0f fix(flux-shard-operator): strip proxy env from cloned helm-controller
A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the
flux-aio helm-controller container. A standalone shard needs no external
egress (source-controller fetches artifacts), so behind an unreachable
corporate proxy it makes a blocking startup HTTPS call that never completes,
the manager never serves /healthz, and the liveness probe crashloops the pod
forever, freezing every HelmRelease carrying a sharding key. flux-aio itself
survives only because it starts once and never restarts.

Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside
the existing localhost/KubePrism sanitisation, so the shard talks to the
in-cluster apiserver directly.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-04 23:07:10 +03:00
Mattia Eleuteri e85501e9f5 fix(backupstrategy-controller): bound each gate check and tolerate an absent Secret
Two defects found in review of the periodic recovery loop.

checkAndLog passed the manager's root context straight into Check, which
makes several sequential API calls. That context is only cancelled at
shutdown, so an API server stalling on any one of them blocked the call
indefinitely. The ticker loop is sequential, so one stuck call stopped
every later check for the life of the pod: the recovery this gate exists
to provide went silent, leaving a stale gauge and no further log line,
with only a restart to bring it back. Each check now runs under a
timeout of half the Period, capped at 30s, so it can never overlap the
next tick.

Check also propagated a NotFound on the projector's source Secret,
although the very next branch treats an empty bucket name as an expected
no-op. Those are the same bootstrap state seen a moment apart, so the
absent Secret logged a check failure on every tick of the whole
bootstrap window for a condition the doc comment frames as graceful.

Both are pinned by tests that fail against the previous behaviour: one
reverts to an error on the absent Secret, the other to a timeout equal
to Period.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-08-03 17:47:36 +02:00
Mattia Eleuteri 51ad226706 fix(backupstrategy-controller): repair lookup-gated backup objects
The default Strategy CRs and the Velero BackupStorageLocation are
Helm-templated behind a `lookup` of the BucketClaim the same chart
creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of
the COSI Secret. When those lookups are empty at install time the objects
were silently skipped, and helm-controller does not re-render a release
whose chart and values did not change (drift detection is off on
operator-generated HelmReleases), so the skip was permanent: clusters ran
for months with only BackupClass cozy-default and no Strategy CRs at all.
That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd
migration, which reads the projected credentials Secret.

Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the
bucket name is resolvable from the projector's source Secret, it checks
that every object cozy-default routes to exists and forces one real Helm
upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing.
Helm remains the objects' only author; the gate only makes sure the render
that produces them actually happens. It cannot be a render-time `fail` in
this chart: the chart is the producer of the Bucket its own lookup reads,
so a failed render would deadlock the condition.

The bucket chart's user-credentials template CAN fail safely — the
BucketAccess that produces the COSI Secret belongs to the parent release
and that release retries forever — so make it fail loudly instead of
skipping, with requireUserCredentials=false as the offline-render escape.

Also correct the chart and doc comments that claimed Flux re-renders on
its interval, and document the two-release manual recovery for clusters
already affected.

Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-08-03 17:16:55 +02:00
Timofei Larkin 95e4faf68e feat(workloadmonitor): bucket metrics endpoint override
The controller discovers where to query SeaweedFS bucket size metrics
via the namespace.cozystack.io/monitoring label, which assumes the
monitoring stack that scrapes SeaweedFS runs in the same cluster. Some
deployments host SeaweedFS and its metrics in a separate cluster, so
add a --seaweedfs-metrics-endpoint flag to cozystack-controller (and a
matching cozystackController.seaweedfsMetricsEndpoint chart value): an
explicit base URL of a Prometheus-compatible query API, with scheme,
host, optional port and path prefix; /api/v1/query is appended. Basic
auth may be embedded as URL userinfo. The flag is validated and
normalized at startup, so a malformed URL fails the rollout rather
than the queries. Empty keeps label-based discovery unchanged.

Bucket sizes feed billing, so metrics failures no longer collapse to
"no size": previously any query failure yielded an empty result map
and the reconciler overwrote Workload resources with it, silently
wiping the recorded sizes. The reconciler now retains the last known
s3-storage-bytes and s3-physical-storage-bytes values, emits a
BucketMetricsUnavailable warning event on the WorkloadMonitor, and
returns the error from Reconcile so it lands in the reconcile error
metrics.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-23 15:35:00 +03:00
Aleksei Sviridkin 6ecb36ffc7 feat(cozystack-controller): extract application CA into a key-free tenant Secret (#3407)
## What this PR does

Adds an engine-agnostic controller that publishes a key-free
`<release>.tenant-ca` Secret (only `ca.crt`) for a managed application,
so a tenant can verify the application's TLS without ever seeing a
private key. The source is declared, not guessed: a chart renders a
namespaced `TenantProjection` sentinel (group `internal.cozystack.io`)
naming the CA Secret to lift; the controller watches the sentinel,
extracts `ca.crt`, and writes the projection owner-referenced to the
sentinel, so garbage collection is native. The projection carries the
`internal.cozystack.io/tenant-ca` label the lineage webhook turns into
tenant visibility. Postgres is the first consumer.

The trust boundary is RBAC: no tenant role grants any verb on
`internal.cozystack.io`, so a tenant cannot forge a sentinel. A
companion `ValidatingAdmissionPolicy` that pins the writer to
helm-controller ships separately as defence in depth.

This is the controller-and-CRD slice, split out from #3299 for focused
review. It is the only piece required for the feature to work; the
render-time helper, the writer policy, the capability-gate, and the
release-prefix validation land as their own PRs.

### Downstream repositories

Walked the trigger map against the diff. This adds an internal CRD and
controller and a sentinel to the Postgres chart; no app is added or
renamed and no `values.schema.json` changes, so the typed provider and
Ansible are unaffected. The one to confirm is the Postgres README edit —
if the website mirrors it, it needs a refresh.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:

### Release note

```release-note
feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added tenant-facing CA certificate projections for PostgreSQL
deployments.
* Tenant secrets now provide only the CA certificate, excluding private
keys.
  * Added support for configuring and monitoring tenant CA projections.

* **Documentation**
* Updated PostgreSQL TLS instructions to retrieve certificates through
tenant secrets.

* **Bug Fixes**
* Improved protection against invalid certificates, secret collisions,
and accidental exposure of private key material.
* Added validation to ensure projected certificates remain synchronized
with their sources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-23 14:14:28 +03:00
Andrey Kolkov 8aeedde088 feat(backups): support PostgreSQL point-in-time recovery (PITR) (#3383)
## What this PR does

Makes PostgreSQL point-in-time recovery (PITR) a genuinely supported,
tested capability for the new `backups.cozystack.io` API. Fixes #2774.

Was **stacked on #3313** (`fix/3300-cnpg-barman-plugin`) for the
`recoveryTime` → CNPG `bootstrap.recovery.recoveryTarget.targetTime`
plumbing; #3313 has since merged and this branch is rebased on `main`.
This PR adds the reliability + verification layer #2774 asked for.

### Changes

- **Precise failure reason for an unreachable target.** The restore
deadline (`spec.options.restoreTimeoutSeconds`, default 30m) stays the
sole authority for "this restore is stuck" — the client waits the full
window either way. What changes is the *diagnosis*: once the deadline
has elapsed, if `recoveryTime` was set and the recovery Cluster never
went healthy, the CNPG driver reads the `full-recovery` pod logs and —
when several distinct attempts exited with PostgreSQL's `FATAL: recovery
ended before configured recovery target was reached` — fails the
RestoreJob with `RecoveryConverged=False` / `Ready=False` (reason
`RecoveryTargetUnreachable`) and an actionable message, instead of a
bare generic timeout. Timing is unchanged; only the reason/message
differ. (For a faster rejection, set a shorter `restoreTimeoutSeconds`.)
Design points:
- **We deliberately do NOT trip early off the FATAL.** A valid near-now
target emits that same FATAL transiently — repeatedly, for many
attempts, on a slow archiver — before it converges, so any earlier trip
would wrongly reject a recoverable restore (observed twice in CI). The
FATAL is used only to *explain* a failure the deadline has already
declared.
- **Health wins unconditionally** — the deadline/classification check
precedes the healthy check, but a recovery that converges within the
window reaches success and is never classified as failed (a live dev7
restore did exactly this).
- **Counting distinct FATAL-ing attempts** (not pod failures) tolerates
a one-off archive-lag miss on a near-now target and ignores transient
recovery-pod crashes (a node blip / brief API-server unreachability
never emits that FATAL).
- Adds the `pods/log` RBAC grant the log read needs (distinct
subresource from `pods`); a `Forbidden` is logged loudly and surfaced in
the failure message rather than silently disabling the classification.
- **e2e** (`examples/backups/postgres/run-all.sh`, wired into the
Chainsaw `postgres-2-backup-roundtrip` suite):
- Step 45 — write `before`, capture server timestamp (µs precision),
write `after`, restore to that timestamp, assert `before` present and
`after` absent.
- Step 46 — a `recoveryTime` an hour ahead must fail with reason
`RecoveryTargetUnreachable` once the deadline elapses (asserts the
*reason*, so it proves the classification fired, not just that a timeout
occurred).
- Chainsaw budget raised 25m → 40m to cover three sequential restores +
the post-deadline classification.
- **Docs + example**: uncommented `45-restorejob-pitr.yaml`; a
`docs/operations/backup-classes.md` PITR section (recoverable window,
discovering earliest/latest restorable time from the backup catalog
since `firstRecoverabilityPoint` is unreliable under the plugin,
archive-lag tolerance, idempotency under GitOps).

### Verification

- `go build` / `go vet` clean; unit tests pass
(`logIndicatesRecoveryTargetUnreachable`, `recoveryUnreachableFromLogs`,
`recoveryPodsToInspect`, nil-Clientset guard, a reconcile-level
success-path test asserting `Succeeded` + `RecoveryConverged=True`, and
— added per review —
`TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable`,
a reconcile-level test driving the deadline branch through all three
sub-paths: FATAL found → reason `RecoveryTargetUnreachable`, FATAL
absent → generic timeout, `Forbidden` log read → generic timeout +
missing-`pods/log`-RBAC note. It fails if the call-site reason is
reverted to the generic one).
- Reviewed via adversarial branch-review; fixes include a critical
missing-RBAC dead-on-arrival, a classification false-positive on
transient recovery-pod crashes, and a sub-second e2e flake.
- **dev7**: backup + WAL archiving + recovery convergence verified live
on the #3313 build. The load-bearing assumption of the classification —
CNPG **accumulates** failed `full-recovery` pods rather than reusing one
— was confirmed (observed 7 simultaneous failed recovery pods). A
fully-green live step-46 run was blocked by intermittent COSI
bucket-provisioning flakiness on dev7; the negative path is covered by
the CI Chainsaw e2e.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. The only user-facing surface is the new
`docs/operations/backup-classes.md` PITR section; the website keeps a
hand-mirrored copy per docs version.

- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: PITR docs mirror warranted (not yet opened; same pending
mirror as #3313's backup-classes changes)
- [x] No other downstream repository is affected by this change

```release-note
feat(backups): PostgreSQL RestoreJob now supports point-in-time recovery (PITR). Set spec.options.recoveryTime (RFC3339) to restore to an exact instant. A recoveryTime past the latest archived WAL still runs to the restore deadline (spec.options.restoreTimeoutSeconds, default 30m), but the failure is now reported with the precise reason RecoveryTargetUnreachable and an actionable message instead of a generic timeout; set a shorter restoreTimeoutSeconds for a faster rejection.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added PostgreSQL point-in-time recovery (PITR) support via RFC3339
`recoveryTime`.
* Introduced a PITR RestoreJob example, plus helper variables and an
unreachable-target scenario.
* **Bug Fixes**
* Improved restore deadline handling with deterministic, log-based
fail-fast classification for unreachable recovery targets, including
clearer conditions.
* **Documentation**
* Added detailed PITR operational documentation (recovery windows,
failure behavior, and GitOps/idempotent behavior).
* Expanded README and example comments for PITR verification and failure
expectations.
* **Tests**
* Extended unit tests and end-to-end chainsaw workflow to cover PITR
success and unreachable failures.
* **Chores**
* Updated RBAC to allow pod log reads; refined cleanup and run scripts
for additional RestoreJob artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-23 14:14:58 +04:00
Andrey KolkovandClaude Opus 4.8 b574e2176f fix(backups): cover Go-driven system-bucket ObjectStore + share the checksum helper
Address review on the S3 request-checksum fix:

- Extend the fix to the platform-managed useSystemBucket=true flow, whose
  ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
  chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
  and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
  barmanSidecarConfiguration(). The platform's own default system bucket is
  SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
  cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.

- Factor the duplicated instanceSidecarConfiguration block (3x across two
  charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
2026-07-22 15:35:32 +04:00
Andrey Kolkov ad3f8b2487 fix(backups): check cluster health before the restore deadline (PITR)
Addresses review CHANGES_REQUESTED on #3383.

CRITICAL: reconcileCNPGRestore checked the restore deadline BEFORE the
cluster-health check, so a reconcile that first observes "deadline elapsed"
AFTER the recovery Cluster already reached a healthy state (a controller
restart or stalled workqueue spanning restoreTimeoutSeconds) would mark an
already-converged restore Failed / RecoveryTargetUnreachable. A false Failed
can then escalate to data loss: a resubmitted RestoreJob's purge guard treats
the recovered Cluster as a stale leftover and deletes it + its PVCs.

Fix: check hasRecovery + cnpgClusterHealthy first and return Succeeded
unconditionally when healthy, regardless of how late the reconcile runs; only
a not-yet-healthy restore reaches the deadline/classification branch. Adds
TestReconcileCNPGRestore_HealthyPastDeadlineSucceeds (StartedAt 72h past the
deadline + healthy Cluster + recoveryTime set; the injected readPodLog fails
the test if the classification path is entered, proving health short-circuits).

MINOR: handle the deferred Close error on the pod-log reader (errcheck).
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 15:31:31 +04:00
Andrey KolkovandClaude Opus 4.8 b5d03f889b test(backups): reconcile-level test for the PITR deadline classification
Review of #3383 found the RecoveryTargetUnreachable classification branch in
reconcileCNPGRestore was only covered through the pure helpers and
recoveryTargetUnreachable directly, never through reconcileCNPGRestore itself —
a mutation swapping the call-site reason back to the generic RestoreFailed kept
the suite green.

Add TestReconcileCNPGRestore_DeadlineClassifiesRecoveryTargetUnreachable: it
backdates StartedAt past the restore deadline with a recoveryTime set and drives
the real reconcile through the readPodLog seam, asserting all three deadline
sub-paths — FATAL present (reason RecoveryTargetUnreachable on both
RecoveryConverged and Ready), FATAL absent (generic timeout, no
RecoveryTargetUnreachable), and a Forbidden log read (generic timeout + the
missing-pods/log-RBAC note). Verified it fails when the call-site reason is
reverted to the generic RestoreFailed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
2026-07-22 13:06:00 +04:00
Andrey KolkovandClaude Opus 4.8 6a1e2d0d06 feat(backups): support PostgreSQL point-in-time recovery (PITR)
Makes RestoreJob.spec.options.recoveryTime a genuinely supported, tested
capability for the new backups.cozystack.io API (issue #2774). The
recoveryTime -> CNPG bootstrap.recovery.recoveryTarget.targetTime plumbing
already existed on the barman-cloud-plugin branch; this adds the reliability
and verification layer the issue asked for:

- Fail-fast for an unreachable target: when a RestoreJob has recoveryTime set
  and the recovery Cluster is not healthy, the driver reads the CNPG
  full-recovery pod logs and, once several distinct attempts have exited with
  PostgreSQL's "recovery ended before configured recovery target was reached"
  FATAL, fails the RestoreJob with conditions RecoveryConverged=False /
  Ready=False (reason RecoveryTargetUnreachable) instead of hanging to the
  restore deadline. Health is checked first, so a converged recovery always
  wins; counting distinct FATAL-ing attempts (not pod failures) tolerates a
  transient archive-lag miss on a near-now target and ignores transient
  recovery-pod crashes. Adds the pods/log RBAC grant the log read requires.

- e2e (examples/backups/postgres/run-all.sh, wired into the Chainsaw
  postgres-2-backup-roundtrip suite): step 45 writes a before marker, captures
  the server timestamp, writes an after marker, restores to that timestamp and
  asserts the before row survived and the after row did not; step 46 submits a
  recoveryTime an hour ahead and asserts it fails fast with reason
  RecoveryTargetUnreachable.

- Working, uncommented example (45-restorejob-pitr.yaml) and a docs section
  (docs/operations/backup-classes.md) covering the recoverable window, how to
  discover the earliest/latest restorable time from the backup catalog, the
  archive-lag tolerance, and idempotency under GitOps.

Refs #2774

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-22 10:40:18 +04:00
Aleksei Sviridkin 37a91f98b8 feat(cozystack-controller): extract application CA into a key-free tenant Secret
Every catalog engine issues a per-release CA, but the object holding
ca.crt usually holds a private key beside it (cert-manager's tls.key,
CloudNativePG's ca.key), so granting a tenant read on it hands over key
material too. Add a controller that publishes the trust anchor as one
canonical, key-free Secret per release, "<release>.tenant-ca", holding
only ca.crt.

A chart declares the anchor by rendering a namespaced TenantProjection
sentinel (internal.cozystack.io/v1alpha1) that names the source Secret
and key. The controller rebuilds ca.crt from the parsed certificate DER
rather than copying the source, refuses any value carrying private key
material, and owns the projection by owner reference so a foreign Secret
at the same name is never overwritten. No tenant RBAC grants any verb on
internal.cozystack.io; a ValidatingAdmissionPolicy pins writes to
helm-controller as defence in depth.

The controller does not grant tenant read access — the lineage webhook
does, from the ApplicationDefinition's spec.secrets. This side stamps the
engine-agnostic internal.cozystack.io/tenant-ca label and forces a
re-admission when those selectors change.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-21 23:59:48 +03:00
myasnikovdaniil 960d853706 feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) (#3273)
## What

Adds typed `waitStrategy` + `healthCheckExprs` fields so a package/app
can declare the Flux helm-controller v1.5.0 kstatus health gate for the
custom resource it wraps — making `HelmRelease.Ready` mean "the CR is
actually healthy" instead of "helm applied the CR". This is the
operator/apiserver foundation of the kstatus epic (#2642); the API
surface was reviewed and approved by the API owner in #3263.

Commits (per the review's "isolated step-0" request):
1. `build(deps)` — the dependency bump (see below).
2. `feat(api)` — the fields, plumbing, tests, and regenerated code.
3. `fix(api)` — match the openapi code-generator pin to the 0.35 runtime
(fixes an aggregated-apiserver crash the bump surfaced).
4. `fix(api)` — declare `OpenAPIModelName` on the apps types so SSA
resolves catalog kinds.
5. `docs(api)` — note that `disableWait` silently defeats
`healthCheckExprs`.

## API surface

Typed fields on **both** HelmRelease-building paths:
- `ApplicationDefinition.spec.release` (apiserver / catalog apps →
`convertApplicationToHelmRelease`)
- `ComponentInstall` (operator / platform packages →
`buildHelmReleaseSpec`)

```go
// WaitStrategy maps to HelmReleaseSpec.WaitStrategy.Name — a deliberate scalar
// simplification of the upstream {name} object. One of poller|legacy.
WaitStrategy string
// HealthCheckExprs maps to HelmReleaseSpec.HealthCheckExprs (the upstream
// kustomize.CustomHealthCheck type verbatim).
HealthCheckExprs []kustomize.CustomHealthCheck
```

`config.ResolveWaitStrategy` is shared by both builders and **couples
the default**: `healthCheckExprs` are only evaluated under the `poller`
strategy, so when expressions are set and no strategy is given, the
generated HelmRelease defaults to `poller`. A package that sets only
`healthCheckExprs` is therefore self-contained, independent of the
controller's global default.

## The dependency bump (commit 1)

The `WaitStrategy` / `kustomize.CustomHealthCheck` /
`HelmReleaseSpec.HealthCheckExprs` types do not exist below
`helm-controller/api` v1.5.0 — the runtime is already v1.5.0 (shipped by
#2602), so this closes a Go-types-vs-runtime skew. It is **not** an
isolated bump: v1.5.x requires **k8s 0.35 + controller-runtime 0.23**,
cascading the k8s stack 0.34 → 0.35 (and the go directive to 1.26).

cozystack replaces `k8s.io/apimachinery` with a fork carrying the
still-open upstream `ConvertToVersion` patch (k/k#135537). That single
commit rebases cleanly onto apimachinery v0.35.0; published at
`cozystack/apimachinery@release-1.35-pr135537`, and `go.mod` pins its
pseudo-version. **The repo builds with zero source changes under the new
stack.**

> Maintainer note: please confirm the `release-1.35-pr135537` fork
branch naming, or fold it into the `cozystack` branch as you prefer —
the pin can be updated to match.

## Not included (conscious YAGNI)

`disableWait` is intentionally omitted. The only known need (the
kubernetes tenant-addon deadlock) was resolved child-side, and
`waitStrategy: legacy` is not equivalent (legacy still waits
Helm-v3-style; `disableWait` skips waiting entirely). Add it later when
a concrete leaf needs a true no-gate escape hatch.

## Tests

- `config.TestResolveWaitStrategy` — the couple-the-default logic
(unset+exprs→poller; explicit legacy honored; unset+no-exprs→nil).
- Operator + apiserver cases mirror each other
(`TestBuildHelmReleaseSpecHealthCheckExprs` /
`TestConvertApplicationToHelmRelease_HealthCheckExprs`).
- `go build ./...` clean, `go vet` clean, `gofmt` clean, codegen
regenerated and committed.

The 4 backupstrategy CRDs regenerate with benign k8s-0.35 PodSpec
description drift (DynamicResourceAllocation alpha→stable, toleration
Lt/Gt, resizePolicy note) — no functional change.

## Follow-up

Phase 3 (per-leaf adoption — postgres/kafka/mongodb/etc. declaring their
CEL via these fields) builds on this. The full CRD→CEL catalog is
maintained in the epic's readiness audit.

Refs #2642

```release-note
feat(api): add `waitStrategy` and `healthCheckExprs` to ApplicationDefinition and ComponentInstall, so a HelmRelease reports Ready only once the custom resource it wraps is actually healthy
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added configurable CEL health-check expressions for application
releases and package components.
* Added `waitStrategy` support (`poller` and `legacy`) with automatic
`poller` selection when health checks are configured and no strategy is
set.
* Propagated these settings into generated Helm releases, affecting
readiness/wait behavior.

* **Documentation**
* Updated Application and PackageSource schemas to include
`healthCheckExprs` and `waitStrategy`.
* Extended related Kubernetes CRD documentation, including projected pod
certificate annotations and workload reference fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-21 17:02:37 +05:00
Andrey KolkovandClaude Fable 5 67db8f7da7 fix(backups): make the barman-cloud serverName stable across every flow
Two defects of the same class — the WAL-archive S3 prefix (serverName)
changing or defaulting differently between components:

Recovery default regression (review blocker): native CNPG defaulted an
absent externalClusters serverName to the entry name (= bootstrap.oldName),
but the barman-cloud plugin defaults it to the NEW cluster's name, so a
legacy-flow restore with bootstrap.serverName unset would read the wrong
S3 prefix (hard failure, or silently restoring another cluster's data).
The chart now always emits serverName, defaulting to bootstrap.oldName.

Live-cluster serverName flip (found by the gated postgres backup e2e on a
real cluster): the chart renders spec.plugins without serverName
(effective prefix = cluster name) while a strategy template may name a
different one; the driver's ForceOwnership patch then flipped the prefix
mid-stream, splitting the WAL archive across two prefixes — the WAL
holding the backup's start was lost between them and the restore failed
with "WAL not found". The driver now preserves the live Cluster's
effective serverName (explicit parameter, or the cluster name when the
plugin is attached without one) and only applies the strategy's value to
a Cluster with no plugin attached; both charts pin serverName explicitly,
and the example strategy no longer sets one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey KolkovandClaude Opus 4.8 268ec29856 test(backups): assert driver-initiated Backup uses method=plugin
Review follow-up (cozystack#3300): ensureCNPGBackup stamps spec.method=plugin +
spec.pluginConfiguration on the postgresql.cnpg.io/Backup it creates — the field
that routes a driver-initiated run through the barman-cloud plugin. If it
regressed to the default (barmanObjectStore) method, CNPG would run the legacy
method against a Cluster that no longer has one and every platform backup would
fail. Add TestEnsureCNPGBackup_UsesPluginMethod asserting method, plugin name,
and cluster ref. (The chart ScheduledBackup path was already covered; the driver
path was not.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey KolkovandClaude Opus 4.8 2212367842 fix(backups): own driver-created ObjectStore + correct stale barman references
Review follow-ups (cozystack#3300):

- Lifecycle/leak: applyClusterPluginBackup now sets an ownerReference from the
  driver-created barmancloud.cnpg.io/ObjectStore to its cnpg.io/Cluster, so
  Kubernetes garbage-collects it when the Cluster is deleted (tenant app
  teardown or the RestoreJob purge). In the platform useSystemBucket flow the
  chart does not render this ObjectStore, so previously it was orphaned in the
  tenant namespace. Drop the now-accurate-only 'delete' verb on objectstores
  from the driver RBAC (GC handles removal; the driver never deletes it) and
  correct the RBAC comment that falsely claimed the purge deletes it.
- Diagnostics/comments: the RestoreJob WAL-archive-deadline failure message and
  several doc comments still told operators to check spec.backup.barmanObjectStore
  / externalClusters[].barmanObjectStore, fields no longer emitted after the
  migration. Point them at the barman-cloud plugin (spec.plugins + ObjectStore,
  externalClusters[].plugin) instead so they don't mislead during an incident.
- Test: TestApplyClusterPluginBackup_PatchesExistingCluster now asserts the
  ObjectStore carries an ownerReference to the Cluster (kind/name/uid).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00
Andrey Kolkov 13ff340277 fix(backups): migrate CNPG backups to the barman-cloud plugin
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-16 21:44:41 +04:00