Commit Graph
680 Commits
Author SHA1 Message Date
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Andrey Kolkov 3b4db69f8e feat(backups): record the RDB snapshot location and size for Redis backups (#5046)
## What this PR does

Builds on the shared S3 probe from #5223 (MariaDB and CNPG), which is on
`main`. Refs #4316.

Backups made by the Redis strategy carry no `Backup.status.artifact` at
all: no URI and no size. This covers both the Redis and Valkey apps,
which share the driver. The snapshot is a single object per backup,
`<namespace>/<app>/<backup>.rdb` (`redisObjectKey`). The driver already
records that key in `driverMetadata`. The default strategy uploads the
object using the S3 coordinates from the backup credentials Secret,
which the controller projects into the application namespace
(`endpoint`, `bucketName`, `region` and the access keys).

When a backup completes, the driver now:
1. reads that projected Secret (`CredentialsConfig.TargetSecretName`) in
the application namespace. The projector stores its `endpoint` without a
scheme, and the strategy script reaches it over https; an endpoint with
a path prefix is refused by minio-go, which leaves the artifact unset;
2. reads the TLS trust from the pod spec of the completed backup Job, in
the order the script applies it: `ca.crt`, then `tls.crt`, from the
Secret mounted as the `s3-ca` volume
(`backupStorage.endpointCASecretName`). Only when there is no `s3-ca`
volume does it honour `S3_INSECURE_SKIP_VERIFY=true`, and only when
every container of the pod sets it, since the probe cannot tell which
container uploaded. The Secret is read at probe time; the default
strategy mounts `s3-ca` only when an endpoint CA is configured, so when
the volume yields no CA now (the Secret is gone or lacks both keys, or
the volume is not a Secret), the probe leaves the artifact unset rather
than trust the system roots or skip verification where the Job may have
verified. The CA is trusted on its own, as `curl --cacert` does, not on
top of the system roots;
3. runs `StatObject` on the recorded key; a response without
`Content-Length`, which minio-go reports as size -1, counts as
unmeasured;
4. records `status.artifact.uri = s3://<bucket>/<key>` and
`status.artifact.sizeBytes` = the object's size. An empty object is
recorded too, but `sizeBytes` is `omitempty`, so on the Backup it reads
like an unmeasured size; since a valid RDB snapshot always carries a
header, a Warning `EmptySnapshot` Event on the BackupJob flags it as a
broken backup.

The artifact is set **only when the object is found**. A custom strategy
that writes elsewhere would otherwise get a URI pointing at an object
that does not exist. On any failure (Secret missing or incomplete,
endpoint invalid, CA unreadable, object not found, TLS error) the
artifact stays absent and the backup still succeeds. The full error is
logged at info, and once the Backup is created a Normal
`ArtifactNotMeasured` Event on the BackupJob gives a fixed reason, also
when a failed existence check skipped the probe; the Event lives in the
application namespace, so it never carries the raw error, which may name
the platform's endpoint and bucket. The probe shares the 10s timeout of
the MariaDB and CNPG probes. The retry check is shared with those
drivers too: a `Backup` that already exists is returned without probing,
a failed existence check creates the Backup without probing, a `Create`
collision is resolved by reading past the cache, and an existing Backup
that cannot be read back is retried with `Ready=False` instead of
failing the BackupJob.

The size is useful because an RDB snapshot is the compressed dataset. It
shows growth, and a snapshot that is suddenly tiny is a broken backup.

Kafka is deliberately left out. Its backup is a topic-metadata export of
a few kilobytes, so the size would say nothing about the data and might
suggest the data itself was backed up.

Tests:
- the stat uses the projected credentials;
- every path that leaves the artifact unset: object missing, Secret
missing, bucket, endpoint, access key or secret key missing, no target
Secret configured, a size the endpoint did not report;
- trust resolution: `ca.crt` wins over `tls.crt` and skip-verify;
`items` remapping of the volume, and a key the `items` do not project is
never trusted; the Redis CA volume mounted next to `s3-ca` is not taken
as the S3 CA; an `s3-ca` volume that yields no CA (Secret gone, Secret
without the keys, a ConfigMap source) is refused with and without
skip-verify; skip-verify applies only on an exact `true` of that
variable in every container, the last entry of a repeated variable
winning;
- the real minio client against a TLS stub: its certificate is trusted
only through the `s3-ca` volume or skip-verify, a CA that did not sign
it is refused even with skip-verify set on the Job, and the shared
client enforces a configured CA with and without the system roots; the
client sends a signed `HEAD` and parses `Content-Length`;
- the Events: an unmeasured artifact or a failed existence check raises
`ArtifactNotMeasured` without the raw error; an empty snapshot is
recorded and raises a Warning `EmptySnapshot`; a measured artifact, a
rejected `Create` and a `Create` collision raise none;
- a reconcile test pinning that the completed Job's pod spec reaches the
probe; a Backup that already exists is returned without a probe or a
`Create`; the `Create` collision is read back from the API server past
the cache.

The suite fails if any guard of the trust lookup (volume name, Secret
source, variable name, every-container rule, refusal of an unreadable
CA), the CA enforcement in the transport, the negative-size check, the
Events, the reason-only message, their place after `Create`, the
empty-snapshot warning, the existing-Backup check, the wiring, any term
of the Secret check, or the API-server read-back is removed.

### Screenshots

N/A — no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is Go-only in
`internal/backupcontroller`, with no API, CRD, chart, value or schema
change. Existing `Backup` fields are now filled for one more driver.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(backups): Redis and Valkey backups now record `status.artifact` with the `s3://` location and the size of the RDB snapshot. If the snapshot cannot be found with the platform backup credentials, the artifact stays unset and a Normal `ArtifactNotMeasured` Event on the BackupJob says why. An empty snapshot raises a Warning `EmptySnapshot` Event.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Redis backups now include artifact location and size when they can be
measured.
* Backups are still created when artifact measurement fails, with status
reporting that the artifact was not measured.
* **Bug Fixes**
* Improved handling of existing backups and temporary read-back failures
to support reliable reconciliation.
* Improved compatibility with S3 endpoints using custom certificate
authorities or configured TLS verification settings.
  * Zero-size snapshots are now reported with a warning.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 15:52:54 +04:00
Aleksei Sviridkin 456473dec1 fix(mariadb): fail backup and restore jobs whose operator Job failed (#5207)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

When the Job behind a mariadb-operator `Backup` or `Restore` fails, the
operator sets `Complete=True` with reason `JobFailed`
(`SetCompleteWithJob` in mariadb-operator 25.10.2). The MariaDB driver
read `Complete=True` alone as success, so a backup whose dump or upload
failed ended `Succeeded` and produced a `Ready` Cozystack `Backup`
pointing at an archive that does not exist, and a failed restore ended
`Succeeded` too. A scheduled `Plan` kept producing such backups.

The driver now treats `Complete=True` as success only with reason
`JobComplete`, and fails the BackupJob or RestoreJob on `JobFailed`
whatever the status, without creating a `Backup`. The failure message
names the operator object and carries the Job's own `Failed` reason and
message, since the operator's condition message is only `Failed` and the
Job's pod, with its logs, is already deleted when the Job fails. A Job
that is missing or cannot be read is named in the message instead. Any
other `Complete` state keeps waiting under the existing deadline, as
before; that includes `Complete=False` with reason `Failed`, which the
operator sets with its own cause (`Error creating Job`, `Backup not
complete`) for a failure outside the Job and then retries, and the
deadline failure now carries that message.

Regression tests drive the backup and restore reconcilers through every
`Complete` status/reason combination, an absent `Complete` condition,
and expired deadlines for an absent condition, an unknown reason and
reason `Failed`. They assert the job phase and message as persisted in
the API rather than on the in-memory object, a requeue while the run is
in progress, and, for backups, that no `Backup` is created on failure.
Separate tests cover the failure message for a Job that cannot be read
and for a Job without a `Failed` condition.

Backups that already exist are not touched. A `Ready` MariaDB `Backup`
created by a failed run before this fix may have no archive of its own.
Restoring from it can fail or select another dump from the shared
prefix.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Backup and restore operations now report failed jobs as failures,
including details from the underlying job when available.
* Operations are marked successful only when the job completes
successfully. Other outcomes continue to be monitored until the existing
deadline, after which failures include available condition details.
  * Failed backups no longer create backup artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-10 11:21:19 +03:00
Andrey Kolkov 1093f390ae feat(backups): warn on the BackupJob when the Redis snapshot is empty
A valid RDB snapshot always carries a header, so a 0-byte object means
the backup is broken. sizeBytes is omitempty, though, so such a Backup
read exactly like one whose size was never measured. The controller
still records the artifact, and now raises a Warning EmptySnapshot
Event on the BackupJob once the Backup is created.

A test now also pins that a retry over an existing Backup returns it
without calling Create.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 12:18:07 +07:00
Andrey Kolkov 2013e624ca fix(backups): raise ArtifactNotMeasured only for a Backup this pass created
The Event was recorded before the Backup was created, so a Create that
then failed left the BackupJob Failed next to an Event saying the
backup succeeded, and a Create that collided with a Backup another pass
had made reported on that pass's Backup. The Event is now recorded once
Create succeeds. It also covers the one unmeasured path that had none:
a failed existence check, which skips the probe.

A test now also pins that a CA Secret key the s3-ca volume's items do
not project is never trusted.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:44:33 +07:00
Andrey Kolkov 6f3d26c491 feat(backups): tell on the BackupJob why a Redis artifact was not recorded
A Backup without status.artifact looked the same whether a custom
strategy wrote elsewhere, the credentials Secret was unusable or the
endpoint CA was gone; only the controller log told them apart. The
probe now returns why it gave up, and the controller records it as a
Normal ArtifactNotMeasured Event on the BackupJob. The Event lives in
the application namespace, so it carries a fixed reason only; the raw
error, which may name the platform's endpoint and bucket, stays in the
log.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:21:46 +07:00
Andrey Kolkov d61a5054a1 fix(backups): refuse to probe a Redis snapshot behind a CA that is gone
The default strategy mounts the s3-ca volume only when the admin
configured an endpoint CA, and the script then trusts that CA alone. When
the volume yields no CA by the time the probe runs, because the Secret
was deleted or emptied after the Job or the volume has another source,
the probe used to fall back to the system roots, trusting more than the
Job may have. It now leaves the artifact unset instead, with or without
skip-verify.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:21:46 +07:00
Andrey Kolkov 43bed8f6d9 fix(backups): keep the Redis probe verifying TLS unless every container skips
The probe read S3_INSECURE_SKIP_VERIFY from any container of the backup
Job, so a custom strategy whose sidecar sets it while the uploading
container verifies made the probe skip verification the upload did
not. The probe cannot tell which container uploaded, so it now skips
only when every container sets the variable to true, the last entry
winning as in the kubelet.

An s3-ca volume of a source other than a Secret yields no CA the probe
can read; it is now treated like a CA Secret that is gone, so it never
turns into skip-verify.

Tests pin the guards that decide the probe's trust: the Redis CA volume
mounted next to the S3 one, a decoy variable set to true, a CA that did
not sign the endpoint, and the AlreadyExists read-back past the cache.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:01:18 +07:00
Andrey Kolkov e03986e8d0 fix(backups): treat an S3 object without a reported size as unmeasured
minio-go returns size -1 when a HEAD answer carries no Content-Length,
which would land in Backup.status.artifact.sizeBytes as -1. The probe
now reports such an object as unmeasured, so the field stays unset.

The shared client's trust settings also gain a test against an endpoint
whose certificate the configured CA did not sign, with the CA alone and
on top of the system roots: nothing pinned that a configured CA is
actually enforced.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 11:01:18 +07:00
Andrey Kolkov 631f6cfc0d fix(backups): mirror the Redis script's CA trust exactly
The script passes the endpoint CA as curl --cacert, which replaces the
default trust store, while the probe trusted the system roots on top of
it. Trust only the CA, as the script does.

A mounted CA Secret that is still there but no longer carries ca.crt or
tls.crt let the probe skip TLS verification when the strategy opts into
it, although the Job may have verified against a key since removed.
Treat a volume that yields no CA now like a vanished Secret and refuse
to skip verification in its place.

Pin the probe timeout and the reconcile's retry of an unreadable Backup
for Redis with tests.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov 50b1a2e5ee test(backups): pin the Redis retry check
Cover the two retry-check paths the Redis driver shares with MariaDB and
CNPG: a failed existence check creates the Backup without probing the
bucket, and an existing Backup that cannot be read back is returned as
errBackupNotReadBack so the reconcile is retried.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov bfea468b6c fix(backups): never skip TLS verification in place of a vanished CA
The Redis probe reads the CA Secret when it runs, not when the backup
Job did. If the pod mounted a CA Secret that has since been deleted and
the strategy also opts into S3_INSECURE_SKIP_VERIFY, the probe skipped
verification against an endpoint the Job may have verified. Refuse to
probe in that case. Without skip-verify a missing CA Secret still falls
back to the system roots, as the script does.

The endpoint comment claimed a bare host; it can carry a path, which the
shared client now refuses.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov 2ff547348d fix(backups): probe the Redis snapshot with the trust its Job used
The default Redis strategy script verifies the S3 endpoint against
ca.crt, then tls.crt, from the Secret mounted as the s3-ca volume
(backupStorage.endpointCASecretName), and only without either honours
S3_INSECURE_SKIP_VERIFY=true. The probe always verified against the
system roots, so on a cluster with a private CA, such as a self-signed
SeaweedFS, every Redis and Valkey backup succeeded with no artifact.

Read the same settings from the pod spec of the completed backup Job, in
the order the script applies them, and pass them to the S3 connection.
The shared transport gains an InsecureSkipVerify switch that is set only
from that explicit opt-in.

The projector always stores the endpoint as a bare host, so the branch
that parsed a scheme out of it was unreachable; drop it and its test.
Cover the endpoint and access-key terms of the Secret completeness check.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
Andrey Kolkov c8cdefc665 feat(backups): record the RDB snapshot location and size for Redis backups
Backups made by the Redis strategy, which also serves Valkey, carried no
Backup.status.artifact at all: neither a URI nor a size. The snapshot is
one object per backup, <namespace>/<app>/<backup>.rdb, whose key the
driver already records, and the default strategy uploads it with the S3
coordinates of the backup credentials Secret the controller projects
into the application namespace.

On completion the driver now stats that object with the same coordinates
and records its s3:// URI and size. The artifact is set only when the
object is found, so a custom strategy that writes elsewhere gets no URI
pointing at a missing object. A Backup that already exists is returned
without probing, as for the MariaDB and CNPG drivers.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-11-10 10:52:45 +07:00
ZverGuy 59cbf4b28e docs(observability-proxy): say what the request timeout bounds
The flag now also bounds the body read and the token and access
reviews, and 0 lifts all of those at once; an operator raising it for
slow store queries needs to know that. The store comment named a
design document the repository does not have.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy d2ef5e031f docs(observability-proxy): describe the proxy as it stands
The package and serve comments described a delivery plan rather than
the code: the command named metrics it does not serve, and serve gave a
plan as the reason to refuse unknown roots before authenticating.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy 4eff1617e5 fix(observability-proxy): bound the token and access reviews
Both reviews ran on the request context, which net/http cancels for a
dropped client only once the body is read. A POST whose body was still
unread therefore waited on a stalled kube-apiserver past every request
limit. The reviews now share the request timeout and answer 503 when it
passes.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy f350b7ed9c test(observability-proxy): observe held slots where they cannot be missed
Polling for a slot that lives only until a 200ms deadline misses it when
the test goroutine is descheduled for longer, as under -race on a busy
runner. The tail and stalled-query tests now read the slot when the
request reaches the store, which the proxy does while holding it; the
slow-body test, which never reaches the store, gets a wider deadline.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy dff23f76e0 test(observability-proxy): spend the rate burst in a loop
Calling allowed twice in one condition reads as a duplicated expression
to staticcheck, which cannot see that each call takes a token.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy f48a94de6d fix(observability-proxy): name the logs time out of range exactly
When both bounds were out of range the answer named one of them at
random, as the check walked a map, and it printed the instant without
its fraction, so a time one nanosecond past the last one vlselect can
hold was shown as a time inside the range. The end is now checked
first and the instant is printed in full; tests pin the inclusive
bound.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy f8ff307f71 fix(observability-proxy): bound only the logs times sent to vlselect
The 1970..2262 check refused times on parsing, including a start that
the window clamps to the maximum range before anything is sent and a
since that only becomes a capped start_offset, so an old start or a
tail resume from before 1970 answered 400 where the documented answer
is to clamp. It had also detached the parseTime doc comment.

The check now applies to the times actually sent, after clamping, and
uses the range vlselect can hold, int64 nanoseconds (1677..2262); 1970
is not a limit of vlselect.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 3722808050 test(observability-proxy): watch the slot instead of probing for it
The slow-body and stalled-client tests checked the slot by sending
requests of their own, which competed for the one slot under test: a
probe that held the slot when the request under test arrived got that
request refused, and the test failed. The tests now read the limiter's
slot count directly.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 109980ffa7 fix(observability-proxy): refuse every empty limit the same way
An empty fields_limit answered 400, but an empty limit or facets cap
fell back to the default, so one malformed parameter got two answers
depending on its name. Every cap now refuses an empty value with 400:
a cap the client sent but left blank is a client bug, and reading it
as the default would hide it.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 15eec1e5b1 fix(observability-proxy): refuse logs times vlselect cannot hold
The proxy sends vlselect RFC3339 times, which it never reads as a
guessed-unit number, but it accepted any RFC3339 year and unix
seconds up to 2286. vlselect keeps times as int64 nanoseconds through
time.UnixNano, which is undefined outside 1678..2262, so such a
window landed anywhere. An absolute time outside 1970..2262 now
answers 400; nothing was logged before 1970.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 429bf5ae76 fix(observability-proxy): answer early refusals without waiting on the body
A request refused before its body was read (401, 403, 405, or a rate
or concurrency 429) had no read deadline yet, and net/http drains up
to 256 KiB of an unread body before it sends the headers. A client
that declared a body and never sent it got no answer, and held the
handler, for as long as it kept the connection; for a second form POST
arriving while the concurrency slot was taken, that was more than five
request timeouts.

The read deadline is now set when the request arrives and lifted once
the body is read, so tails are not cut. A refusal sent with the body
still unread closes the connection instead of draining it, so it is
answered at once.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 5db02ab47e test(observability-proxy): pin the concurrency Retry-After value
The test only checked that a Retry-After was present, while the rate
limit test pins its value, so a change to the promised hint passed.
It now expects the literal value. The note on the empty fields_limit
case now gives the reason it is refused.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 1004f2f2d8 test(observability-proxy): pin the refused forms of the bearer header
Parsing the scheme with a case-insensitive split changed which header
forms reach the token review; a bare scheme, an empty token and a
scheme run into the token must still answer 401.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 38f9837f14 fix(observability-proxy): cap the hits groups when fields_limit is absent
The fields_limit cap applied only when the client sent the parameter.
vlselect returns every group when it is absent, as it does for 0, so
leaving it out bypassed the cap. The capped value is now always sent.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 1ef10537f6 fix(observability-proxy): send Retry-After with the concurrency 429
Only the rate-limit 429 said when to retry. A concurrency or tail 429
came without it, so a client had no hint and could retry at once in a
loop. It now carries a one-second hint; when a slot frees depends on
the user's other requests, which the proxy cannot predict.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 650a99ff27 fix(observability-proxy): accept the bearer scheme in any case
The authorization scheme is case-insensitive, and kube-apiserver
accepts "bearer" as well as "Bearer", so a client that works against
the API server got 401 from the proxy with the same header.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:07 +07:00
ZverGuy 95ed8134e4 fix(observability-proxy): refuse an empty fields_limit on hits
An empty fields_limit fell back to the default of 0 and reached
vlselect as fields_limit=0, instead of being refused like any other
malformed limit.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 7dc6ac3687 fix(observability-proxy): read a leftover request body under the deadline
ParseForm reads no body on a GET or on a POST that is not a form. When
such a request declared a body and never sent it, net/http drained it
after the handler flushed the headers, with no read deadline, so the
request held its slot for as long as the client kept the connection.

The handler now reads the rest of the body itself, under the same
deadline and size cap as the form. Draining was chosen over refusing a
declared body so that clients which send one stay working. A body
over the cap now answers 413 instead of a generic 400.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy b1108e836a fix(observability-proxy): refuse to resolve a store before the cache syncs
An absent namespace resolves to the root store, and before the
informer's first list every namespace is absent, so a request served
in that window read a labeled namespace's logs from the wrong store.
Readiness keeps the Service from routing there, but a client that
addresses the pod directly is not stopped by it.

The resolver now fails while the cache is unsynced, which answers
503 like any other resolution error.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy cd04c3e6c4 fix(observability-proxy): count the aligned buckets in the logs step
vlselect widens the window to step boundaries and returns a
zero-filled bucket for each one, so a start off a boundary yields one
bucket more than the span divided by the step. Sizing the step from
the unaligned span let hits and stats_query_range return MaxBuckets+1
buckets.

The minimum step now spans MaxBuckets-1 steps, and the smallest
accepted --logs-max-buckets rises to 4 so that this step still fits
the third-of-range cap that keeps alignment inside the window.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy d3e6fd6aa8 fix(observability-proxy): keep the rate-limit buckets under their bound
When the bucket map was full and the sweep freed nothing, the new
user's bucket was added anyway, so the map grew past its bound and
every further newcomer paid a full scan under the limiter lock.

A newcomer is now refused with 429 until the first bucket refills,
and the Retry-After tells it when. Resetting the map instead would
hand every throttled user a fresh burst.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy bb073eec1b fix(observability-proxy): do not cache reviews that failed to decide
A TokenReview whose authenticator failed comes back unauthenticated
with status.error set, and a SubjectAccessReview whose authorizer
failed comes back with an evaluation error and no decision. Both were
cached as refusals, so a valid caller stayed locked out for the cache
TTL after the outage ended.

Such answers are now refused for the current request only. A review
that failed to decide still answers 401 or 403 rather than 503: RBAC
sets an evaluation error for any binding to a missing role, which a
retryable status would turn into a permanent 503 for that user.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 0dc8a49c19 fix(observability-proxy): cut the connection when a log answer breaks off
Once the status line has gone out, a read error from the store left
the handler returning normally, so the chunked body ended cleanly and
the client took a partial answer for a complete one.

The handler now aborts the response in that case. The request is
counted before the abort, because the abort skips the rest of the
handler.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 8b8b81013c fix(observability-proxy): bound the wait for tail response headers
The tail idle timer started only once the store had sent its headers,
and a tail request carries no deadline, so a store that accepted the
request and stayed silent held the tail slot until the client left.

The timer now starts before the request, and the same timer bounds the
header wait and the gap between lines. A store that never answers
gets 504, like a non-tail request past its deadline.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy ddc37f9c4f fix(observability-proxy): put a deadline on writes to the client
A client that stopped reading kept its slot for good: the first flush
ran before the tail idle timer started, and cancelling the upstream
request does not interrupt a write that is blocked on the client.

Every response now carries a write deadline. A tail renews it by the
idle timeout before each line, so a tail that keeps receiving stays
open while a stalled reader is cut and frees its slot; any other
response must be written by the request deadline.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy f81aaf4d25 fix(observability-proxy): admit a request before reading its body
The form was parsed before the concurrency slot was taken, and the
listener sets no body-read deadline, so a client that never finished a
POST body held a handler outside every limit for as long as it liked.
Tenant workloads reach the Service directly, so nginx buffering does
not cover this path.

The slot is now taken first and the body is read under the request
timeout; a body that does not arrive in time answers 408 and frees the
slot. The deadline is lifted once the body is read, because the
server's background read would take a later expiry for a client that
went away and cancel a live tail.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 4101dac9b2 test(observability-proxy): pin gzip on non-tail log requests
Only tails opt out of the transport's gzip negotiation; bounded
requests keep it on purpose. Assert that side too, so a change that
applies identity to every request fails a test instead of silently
dropping compression between the proxy and the store.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy e6e3466b7d fix(observability-proxy): open live tails without waiting for a line
Go's transport asks for gzip on its own when a request sets no
Accept-Encoding. vlselect answers a gzip-negotiated tail through a gzip
writer that holds even the response headers until the first line, so the
proxy could not forward its 200 and the console kept showing a
connecting state for quiet applications; buffered lines were also
delayed.

The tail request now sends Accept-Encoding: identity, which also turns
off the transport's transparent gzip. Other endpoints keep it: they are
bounded responses read to the end, compression saves bandwidth between
the proxy and the store, and the transport decompresses before the body
is copied to the client.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy f19363ec62 fix(observability-proxy): keep Retry-After positive at tiny rates
The wait until the next token was converted from float to Duration
without a range check, so a rate limit below about 1e-10 requests per
second produced a negative Retry-After on amd64. The wait now saturates
at the largest Duration.

The earlier 429 change said range, step, bucket and limit violations
keep 422; on the logs root they are clamped or answer 400, and nothing
answers 422.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 97a536b89a fix(observability-proxy): refuse durations past the int64 range
parseDuration summed duration parts without an overflow check and
converted out-of-range floats to Duration, which Go leaves
implementation-defined. A start_offset such as
99999999999h99999999999h1s wrapped to 999ms and was accepted, and a
huge step saturated on some architectures and wrapped on others. Such
values now answer 400 like any other invalid duration.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
scooby87 10e0ef9ece fix(postgres): fail a backup job as soon as its cnpg Backup fails (#5211)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Part of #5201. The remaining part, a bound on a Backup that stays
`pending`, is in #5209.

A Postgres BackupJob whose `cnpg.io/Backup` had failed stayed `Running`
for 30 minutes with reason `CNPGBackupTransientFailure`, on the premise
that CloudNativePG may move a Backup out of the failed phase. It does
not: in CloudNativePG 1.30.0 the backup controller returns early on any
Backup already `failed` or `completed`, so the wait only delayed the
verdict. The BackupJob now fails as soon as the Backup is `failed`, with
the error CloudNativePG recorded. When that error is only the plugin's
exit status, the message points to the `plugin-barman-cloud` sidecar
log, where the storage error (`AccessDenied`, a connect timeout) is.

A Backup in any other phase keeps the BackupJob waiting as before, but
the BackupJob now carries a `Ready=False` / `WaitingForCNPGBackup`
condition naming the Backup and its phase instead of no condition at
all. These phases stay uncapped on purpose: CloudNativePG keeps a
healthy Backup `pending` while another backup of the same Cluster runs
and while it restarts one on another instance, so a fixed deadline here
would fail healthy backups, in particular scheduled ones that queue
behind a long previous run. #5209 tracks bounding a Backup that is stuck
rather than queued, together with the Plan concurrency that makes
queueing common.

A table-driven test drives the reconciler through each Backup phase and
asserts the job phase, the message, the persisted condition, that an
unchanged phase does not rewrite the BackupJob status, and that only a
completed Backup produces a Cozystack `Backup`.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(postgres): a Postgres BackupJob now fails as soon as its CloudNativePG Backup fails, with the recorded error, instead of after 30 minutes, and a BackupJob waiting on a CloudNativePG Backup reports a WaitingForCNPGBackup condition with the Backup's phase.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* CNPG backups that enter a failed state now fail their BackupJob
immediately, with the recorded error shown when available.
* BackupJobs waiting for a CNPG backup to finish remain in a waiting
state, even after the backup deadline; the deadline applies while
waiting for the backup to be created.
* Waiting status now includes a clear reason, and failure messages
provide a sidecar-log hint when relevant.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 18:17:03 +03:00
Aleksei Sviridkin 9e8e10c307 fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does

Refs #5205.

The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.

The gate now also forces a render when a copy is stale:

- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.

Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.

What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.

Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.

Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:02:42 +03:00
ZverGuy 163b079845 fix(observability-proxy): answer 429 to rate and concurrency limits
Clients such as Grafana and browser retry logic back off on 429 Too
Many Requests, not on 422, which they treat as a final rejection of
the query. The per-user rate limit, the per-user concurrency limit and
the open-tail caps are about the caller's pace, not the query, so they
now answer 429. The rate limiter knows when its bucket refills, so that
answer carries Retry-After; the concurrency caps cannot know when a
slot frees and omit it. Range, step, bucket and limit violations keep
422.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy dd96a415a9 test(observability-proxy): make every bucket-cap case need the rounding
Two of the three windows divided into whole milliseconds, so they
passed with the old truncating step and guarded nothing. Each window
now leaves a sub-millisecond step remainder.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 28b391ff17 fix(observability-proxy): round the step up to whole milliseconds
The step is sent to vlselect in milliseconds and was truncated, so a
window that the bucket cap does not divide evenly got one bucket more
than --logs-max-buckets allows. Round it up before the window is
shrunk to fit the step, so both caps still hold.

Bare-seconds steps and start offsets are now tested on the accepting
side too: only the refused forms were, and disabling the digit branch
passed the suite.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 4e8071aa2b fix(observability-proxy): accept bare seconds only as plain digits
Bare seconds were parsed with ParseFloat, which also takes signs,
exponents, Inf and NaN. Such a start fell through to the relative
duration form and silently moved the window, so 1.7937864e9 meant
1970 rather than an error, and NaN converted differently per CPU
architecture. Only digits with an optional fraction are taken now;
everything else is a 400.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00