mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
54b76fcdf1ff61402a695eb790381dbf1428e46f
838
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b24a9b4cb9 |
Merge branch 'main' into feat/observability-proxy-exported-namespace
Resolve the overlap with the per-application roots: the namespace root keeps its platform exporter branch, the application root still replaces the whole scope with its own selectors, and the label APIs mark the repeated scope so that replacement reaches match[] as well. Assisted-by: LLM Signed-off-by: Maxim Belyy <maximbel2003@gmail.com> |
||
|
|
4f7eb63447 |
feat(observability-proxy): serve per-application logs and metrics (#5220)
## What this PR does Depends on #5219. This PR is stacked on that branch; the description covers only what it adds on top. Adds the per-application roots to the observability proxy, as described in the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94): ```text /apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint> /apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint> ``` This lets the console show the logs and resource consumption of one application rather than of the whole namespace. What is added: - Plural-to-Kind mapping from an informer over `ApplicationDefinitions`. The Kind is looked up only after the `SubjectAccessReview` (`get <plural>/log` or `get <plural>/metrics`), so an unauthenticated caller cannot probe which plurals exist. An unknown plural answers `404`; an unsynced cache or a plural declared by two kinds answers `503`. - Logs: a second `extra_stream_filters` value with the application Kind and name, on top of the unchanged namespace filter. VictoriaLogs ANDs the two, on every endpoint including `tail`. - Metrics: the forced `extra_filters[]` becomes an OR of three branches, each still bound to the namespace: 1. series carrying the application identity labels; 2. series of the application's pods, found through `kube_pod_labels`; 3. series of the application's PVCs, found through `kube_persistentvolumeclaim_labels`. - The pod and PVC lookups cap the number of names (`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap answers `422`. A failed pod or PVC lookup fails the request (`502`, or `504` on the deadline) rather than answer without the application's series, and so does a lookup that vmselect marks `isPartial`, since a storage node that did not answer leaves names out (a partial list already over the cap still answers `422`); vmselect's series-limit refusal of a lookup is passed through as `422`. A query that reads further back than `--metrics-app-lookback` (default 720h, chart value `observabilityProxy.metrics.appLookback`), or by an unknown amount, and a query whose samples lie at a time the lookup cannot cover answer `422` on the application root; this includes every query with an `@` modifier, `WITH` with a negative offset, and offsets that are not plain durations. The namespace root still serves them. `rollup_candlestick`, a backslash or a `#` comment outside a string, an offset directly followed by a minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery step that is not a plain duration are refused as well. The lookup reaches past the evaluation time by any negative offset and by two steps per subquery level. - Application routes require read access to the namespace root as well as the application subresource; a grant narrowed with `resourceNames` alone answers `403`. - The proxy's ClusterRole gains `list`/`watch` on `cozystack.io/applicationdefinitions`. No tenant RBAC change is needed: the `view` role already allows `get` on every `apps.cozystack.io` resource and subresource. - `/readyz` now also waits for the `ApplicationDefinitions` cache, so a missing CRD or RBAC rule keeps the whole proxy unready, namespace roots included. #### Data comes from the identity labels The application roots find data only through identity labels added by three PRs that are now in main: - #5215 `feat(monitoring-agents)`: export PVC labels from kube-state-metrics (PVC branch). - #5216 `feat(monitoring)`: stamp application identity on scraped series (identity branch). - #5217 `feat(monitoring-agents)`: keep application identity in container log streams (logs root). Merge after #5219. The pod branch does not depend on them, because `kube_pod_labels` already exports all pod labels. Logs and series written before those changes rolled out carry no identity and are reachable only through the namespace root. #### Security condition: a convenience filter, not an isolation boundary The application roots narrow the namespace roots; they do not isolate applications from each other. Each application route therefore requires two grants: the application subresource (`get <plural>/log` or `get <plural>/metrics`) and read access to the namespace root (`list cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant narrowed with `resourceNames` alone is refused. Every tenant role that can read applications already has both. The application scope cannot isolate yet because: 1. The lineage webhook keeps `apps.cozystack.io/application.*` labels that a user sets on a pod or PVC that has no application owner, so such an object can claim another application's identity. 2. The webhook's `objectSelector` skips any object created with `internal.cozystack.io/managed-by-cozystack`, pods included, so forged labels on such an object are never examined. 3. The pod and PVC branches match by name over the lookup window, so an object that reuses the name of another application's recently deleted pod or PVC pulls that object's older series into scope. The proposed fix (a separate change to the lineage webhook) is to strip the identity labels when no owner resolves, and to stop skipping CREATE requests for objects carrying the managed-by label. #### Other known gaps - PVCs created from a StatefulSet's `volumeClaimTemplates` usually get no lineage labels, so their volume series are not in the application scope. - ingress-nginx series intentionally carry no application identity, so they are reachable only through the namespace root. - A request for an application that does not exist answers `200` with an empty result rather than `404`: the proxy checks authorization, not existence. Nothing leaks; whether a `404` is wanted is open for review. #### How it was tested - `go test -race` and `go vet` for the proxy packages, including a narrowed-grant test (access to application `a` only): every application path for `b` answers `403` and no request reaches the store. Name escaping in the pod and PVC branches is covered for regex metacharacters and quotes. - helm-unittest for `observability-proxy` (new RBAC rule). - Verified on a test cluster, with the three identity PRs above applied: the application logs root of a Redis instance returned only that instance's lines and none from a neighbouring pod in the same namespace; its application metrics root returned only the Redis pods' series. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff on top of #5219 is confined to the proxy's Go code and its own chart RBAC; it reads `ApplicationDefinitions` but does not change their semantics or the CRD. The security condition above may deserve a note in the design proposals in `cozystack/community`, which is not a trigger-map item. ### Release note ```release-note feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added a tenant-aware observability proxy for logs and metrics, with namespace and application scoping, access checks, and per-user request limits. * Enabled the proxy in the system package bundle and routed console log and metrics requests through it. * Added live log streaming and health, readiness, and metrics endpoints. * **Bug Fixes** * Invalid or excessive query parameters are rejected; requests with uncertain application-metric scope are refused rather than returning incomplete results. * **Tests** * Added end-to-end and integration coverage for tenant isolation, console routing, access controls, and log streaming. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
526f08e17d |
fix(mariadb): restore the dump a Backup was taken as (#5227)
## What this PR does A RestoreJob that names a MariaDB `Backup` restored the newest dump of the application, not the dump of that `Backup`. The driver created the operator `Restore` with only `backupRef`, and mariadb-operator then restores the dump nearest to the moment the restore starts, from the prefix every run of the app shares. Restoring an older `Backup` brought back the latest data, and when its own dump was already gone it silently restored another one. The `Backup` now records the dump its operator `Backup` wrote, the file name in `k8s.mariadb.com/dump-file` and the timestamp from that name in `k8s.mariadb.com/dump-time`. They come from the ownership-proven match the size probe already makes, and nothing is recorded when ownership is not proven. A restore first checks with a bounded S3 listing of the operator `Backup`'s own storage that the dump still exists, then sets `targetRecoveryTime` on the operator `Restore` to the dump's own timestamp, which mariadb-operator matches exactly. If the dump is gone the RestoreJob fails and names it. If the check cannot be made it waits with `Ready=False/DumpCheckPending` until the restore deadline and then fails with the cause. `Backup`s taken before this stay restorable. The restore makes the same ownership-proven match against their operator `Backup` and pins the dump when it is provable, otherwise it restores the newest dump as before and a `BestEffortDump` warning event on the RestoreJob says so. One case is not covered: the `Backup`'s own dump is gone, a neighbour's dump from within the minute of slack is still there and its operator `Backup` was deleted, so that dump is taken as the match. Nothing in the platform deletes operator `Backup`s, it needs someone to delete one by hand. A `Backup` made of an operator `Backup` whose Job failed (`Complete` with reason `JobFailed`) has no dump, so its RestoreJob now fails instead of restoring another run's dump. Stacked on https://github.com/cozystack/cozystack/pull/5223 and targets its branch until that merges, only the last three commits belong here. Checked on a test cluster with mariadb-operator 25.10.2. An operator `Restore` of an older `Backup` with `targetRecoveryTime` set to its dump timestamp restored exactly that dump (`obtained target backup` names it and a row inserted after it is absent from the target), the same `Restore` without the target restored the newest one. Against the real bucket and operator `Backup`s the recording probe attributed to each `Backup` the dump its backup pod logged, the existence check passed for those dumps and refused a pruned one, legacy `Backup`s with a provable dump got pinned while back-to-back runs fell back with the warning, and the two failed runs on that cluster were refused. The controller itself was not run on the cluster, only its code driven by a test against the real bucket. Fixes https://github.com/cozystack/cozystack/issues/5196 ### Screenshots Not a UI change. ### Downstream repositories The diff is confined to the backup controller's Go code. It changes no CRD or API type (`driverMetadata` is an opaque map), no chart, values, package layout or `hack/` script, so no entry of the trigger map applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(mariadb): a RestoreJob now restores the dump of the Backup it names instead of the newest dump of the application. It fails with a clear message when that dump no longer exists in storage, or when the Backup was made of a failed backup run. Backups taken before this release are restored by the one dump that can be matched to them, or else, as before, by the newest dump, with a BestEffortDump warning event on the RestoreJob. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * MariaDB backups now record the selected dump’s filename and timestamp, allowing restores to target the corresponding recovery time. * For older backups without dump metadata, restores use a uniquely matched dump when possible or fall back to the latest dump with a warning. * Restore processing verifies recorded dumps, retries temporary storage checks, and reports failures for invalid or missing recorded dumps. * **Bug Fixes** * Deleting a MariaDB backup now removes its associated dump and operator backup when ownership is verified and no restore is active. Cleanup avoids deleting artifacts when it cannot safely identify them. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6f8171db89 |
fix(workloadmonitor): report a bucket's workloads not ready until its claim is provisioned (#5096)
## What this PR does A Bucket app reports WorkloadsReady=True while its BucketClaim is never provisioned. On a v1.6.5 cluster a tenant got a BucketClaim for bucket class `tenant-root`, the root tenant there runs no SeaweedFS, and for three days the claim had no status while the objectstorage controller kept emitting `FailedCreateBucket` with `bucketclasses.objectstorage.k8s.io "tenant-root" not found`. The Bucket app showed Ready and WorkloadsReady the whole time. The WorkloadMonitor controller already writes a Workload per BucketClaim with `Operational` set from `status.bucketReady`, but the monitor itself counts only pods. The bucket chart renders its monitor with `replicas: 0` and `minReplicas: 0`, so the monitor stays operational whatever happens to the claim. The monitor now names every selected BucketClaim that is not provisioned in `status.message`, sets `status.reason` to the new `BucketClaimNotReady` and turns `Operational` false, the same way it already reports a DataVolume that is not ready. A DataVolume verdict keeps the fields when both apply. BucketClaims are listed on every pass, so a stored BucketClaim verdict is recomputed rather than kept like an unread DataVolume verdict, and it clears once the claim is provisioned. Any monitor whose selector picks a BucketClaim follows its readiness, which is what the Workload objects already say. The Application then shows WorkloadsReady=False with that message and reason. Its Ready condition keeps following the HelmRelease, as `pkg/registry/apps/application/rest.go` intends. Three new tests cover a provisioned claim, a claim not provisioned with and without a class, a claim outside the selector, the verdict clearing after provisioning with and without a DataVolume reader, and the DataVolume verdict winning over the BucketClaim one. Against the controller on main the four not-provisioned and clearing cases fail and the rest pass. `go test ./internal/controller/...` and `go vet` pass. `make generate` does not run on my machine, gengo fails to parse the go1.27.1 standard library, so I ran the controller-gen `object` and `crd` steps of `hack/update-codegen.sh` for `./api/v1alpha1/...` directly. The deepcopy file is unchanged and the generated WorkloadMonitor CRD matches `packages/system/cozystack-controller/definitions/cozystack.io_workloadmonitors.yaml`. I have not run the change on a live cluster. #5083 changes the Workload names of the same BucketClaims and does not touch the monitor status, so the two do not overlap. ### Screenshots Not a UI change. ### Downstream repositories The controller, one exported constant and its tests. No CRD schema, values key, default, telemetry metric or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(workloadmonitor): a Bucket app reports WorkloadsReady=False with reason BucketClaimNotReady while its BucketClaim is not provisioned, instead of WorkloadsReady=True. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Workload monitors now report when a selected bucket claim is not provisioned and mark the monitor as non-operational. Status messages include the bucket class when available and list multiple unprovisioned claims in sorted order. * **Bug Fixes** * The bucket-claim status clears when the claim is provisioned or no longer selected. When DataVolume information is available, its status takes precedence over the bucket-claim status. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f12300943d |
chore: clear golangci-lint findings and handle the errors they flagged (#5306)
## What this PR does
This clears the golangci-lint findings on main and fixes the errors
several of them pointed at. With the default linter set (errcheck,
govet, ineffassign, staticcheck, unused), main had 197 findings when
this branch started. Rebased on main at
|
||
|
|
5d453c4d23 |
test(backups): give the fake S3 bucket a StatObject
main added StatObject to the object lister interface, so the fake bucket the MariaDB cleanup tests use has to implement it to compile. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
8a37a70681 |
Merge branch 'main' into fix/mariadb-restore-dump
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
03a2edf3db |
Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
17bbca2088 |
feat(observability-proxy): serve tenant namespace metrics (#5219)
## What this PR does Depends on #5214. This PR is stacked on that branch; the description covers only what it adds on top. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. Adds the namespace metrics root to the observability proxy, as described in the design proposal [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94). A tenant user can query the metrics of their own namespace from VictoriaMetrics with their Kubernetes credentials, without seeing other tenants' series. What is added: - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/{query,query_range,series,labels,label/<name>/values}`, GET and POST. Anything else under the root answers `404`. - Authorization by one `SubjectAccessReview` (`list` on `cozystack.io/metrics` in the namespace); the aggregated tenant `view` role gains that rule. - The upstream request is rebuilt from an allowlist (`query`, `start`, `end`, `step`, `time`, `match[]`, `timeout`). The scope is a single forced `extra_filters[]={namespace="<ns>"}`; client-supplied `extra_filters`, `extra_filters[]` and `extra_label` are dropped, because vmselect would OR them with the forced filter. The query itself is not rewritten. - Label endpoints always carry a `match[]` (the namespace scope when the client sent none), so they stay scoped even if vmselect's `-search.ignoreExtraFiltersAtLabelsAPI` is ever enabled. - Cost limits: maximum range (30 days), maximum points (11000), and a clamped `timeout` (30 seconds); query-shape limits answer `422`, the concurrency limit answers `429`. Times are accepted as unix seconds or RFC3339; durations are rounded up to whole milliseconds before limits are checked. - Store selection between the short-term and long-term VictoriaMetrics stores, configured by new chart values (`observabilityProxy.metrics`). #### Store selection: a known limitation and an open question The proxy has to send each query to the short-term store (3 days of retention, finer deduplication) or the long-term store. The choice depends on how far back the query reads, which is not only the request window but also the windows, subquery steps and offsets inside the expression. The current implementation estimates that lookback with a conservative scan of the expression text, not with a parser. It sums every `[...]` window and subquery step, every positive `offset`, and allowances for implicit subqueries and vmselect's default step and latency offset. Anything it cannot read with certainty (`@`, `WITH` templates, step multiples such as `5i`, template variables, unusual duration syntax) goes to the long-term store. The query is forwarded unchanged, so the estimate affects only the store choice, never the scope. This is a heuristic. It is designed to err only towards the long-term store, which holds the same recent data at coarser deduplication, so a wrong guess returns coarser data rather than missing data in most cases. A very short `rate` window sent to the long-term store can still come back empty. Two review rounds each found new expressions the scan underestimated, which were then fixed; there may be more. Open question for reviewers: should the scan be replaced by parsing the expression with `github.com/VictoriaMetrics/metricsql` (the library vmselect itself uses), only for the store choice? That gives an exact lookback at the cost of a new Go dependency. The namespace scope would still be enforced by vmselect through the single `extra_filters[]`, so the isolation design does not change either way. A user-visible consequence of counting the lookback: a range whose start sits right at the short-term retention edge (for example "last 3 days") now goes to the long-term store. #### How it was tested - `go test -race` and `go vet` for the proxy packages; table tests for the lookback estimate, including mutation checks of the `@` and unparseable-window cases. - helm-unittest for `observability-proxy` (new args and the tenant role rule). - Verified on a test cluster: `401` and `403` where expected; results contain only the caller's namespace; a query naming another tenant's namespace and a client-supplied `extra_filters[]` return nothing extra; `query_range`, `label/<name>/values` and `series` are scoped; an admin endpoint answers `404`. - Not verified: the limits under load, and the store choice on windows longer than 3 days (the test cluster had less than an hour of data). #### Known limitations - Relative times (`start=-1h`, `end=now`) are refused with `400`, while the logs root accepts durations. The console sends unix seconds, so it is not affected. - The proxy bounds the evaluation window, the step, the number of points of `query_range` and the timeout, not how far an expression reads: windows and offsets inside the expression (`up[10y]`, `x[30d:1ms]`, `offset 20d`) are bounded only by the store's retention and vmselect's own limits. - Times and durations are sent in a form vmselect parses exactly (RFC3339 with milliseconds for times, durations that truncate to the checked milliseconds); times outside 1970..2262 are refused with `400`. A test repeats vmselect's parsing of the exact strings the proxy sends. - `--metrics-max-range` defaults to 30 days while the longterm store keeps 14, so the older part of a long range comes back empty rather than refused. The two should be aligned or documented. - The store estimate errs towards longterm: deeply nested expressions with a large step can go to longterm although shortterm holds the data, and longterm deduplicates at 5 minutes, so short windows such as `rate(x[1m])` can come back sparse there. - The `namespace` label is only as trustworthy as the writer. Any pod in a child tenant may write to the vminsert of every ancestor tenant (`packages/apps/tenant/templates/networkpolicy.yaml`), and vminsert does not authenticate writers, so a child tenant can write series labelled with another tenant's namespace. Through this proxy the owner of that namespace would see those series as its own, and a flood of them could push its queries over the series limit. Reading another tenant's data is not possible this way. The write path predates this PR and already affects the root tenant's dashboards and alerts; closing it needs the write side to mark untrusted series, for example a vmauth in front of vminsert that forces an `extra_label` the proxy then excludes. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff on top of #5214 is confined to the proxy's Go code and its own chart under `packages/system/observability-proxy` (no platform values, no package under `packages/apps/`, no CRD, no `hack/` change). A user-facing page for the metrics API on the website would be useful but is not required by the trigger map. ### Release note ```release-note feat(observability-proxy): let tenant users query their namespace metrics from VictoriaMetrics with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/`. Access requires the tenant `view` role or higher. Ranges are limited to 30 days and 11000 points. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * The observability proxy now supports tenant-scoped metrics queries alongside logs. * Metrics requests are routed to short- or long-term stores based on their time range, with configurable retention and query limits. * Metrics access is subject to tenant authorization, request validation, and concurrency limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
7c46e1e384 |
fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does The barman-cloud sidecar of Postgres backups is deliberately left without a CPU limit, so that a throttled sidecar slows WAL archiving down instead of failing it. In a tenant with `resourceQuotas` set that intent does not hold: `packages/apps/tenant/templates/quota.yaml` creates the `tenant-range-limits` LimitRange, whose default CPU limit is `250m`, and the sidecar receives it. Full backups and WAL segments are then gzip-compressed on a quarter of a core. On a production tenant the first full backup of a 1.6 GB database took 30 minutes, uploading at about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small database on the same destination finished in 35 seconds. This sets an explicit `limits.cpu: "1"` on the sidecar, in the two places that build its configuration: `barmanSidecarConfiguration` in the backup driver (platform-managed ObjectStores) and `cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the postgres and keycloak charts). This mirrors how the memory request and limit there already override the LimitRange's 128Mi default. One core removes the 250m ceiling without letting the sidecar starve the instance. Inside a tenant it counts against the `limits.cpu` quota, one core per instance pod; the comments now say so. The sidecar is injected when the instance pod is created, so existing instances pick the new limit up at their next restart. Tests: the Go test that asserted the absence of a CPU limit now asserts one core, and so does `packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail with the implementation reverted and pass with it. `go test ./internal/backupcontroller/`, `helm unittest packages/apps/postgres` (220) and `helm unittest packages/system/keycloak` (121) pass. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff (the backup driver, `cozy-lib`'s barman template, two tests): no app, value, schema, default, CRD, annotation or tooling change, so none of the listed repositories restates anything touched here. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart. ``` |
||
|
|
e8c8fa3ec7 |
Merge branch 'main' into fix/workloadmonitor-bucketclaim-ready
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6cab97f010 |
style: use tagged switches and drop unused test assignments
Use tagged switches where staticcheck suggests them and drop two test assignments whose values are never read. Behaviour is unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6f2d6ed17b |
fix(sse): refuse streams that open while the server shuts down
Shutdown stopped the clients registered at that moment and then shut the HTTP server down. A stream accepted in between registered after the stop loop, so nothing ended it, and http.Server.Shutdown waited for its handler. The kube-ovn plunger calls Shutdown without a deadline, so such a stream delayed stopping until the manager's graceful-shutdown timeout and then failed it. Shutdown now marks the server as closing under the same lock, and a stream that opens after that is answered 503 without being registered. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
40d340a3fc |
fix(sse): wait for the pump before returning
When a client disconnected, the handler stopped its pump and returned at once, while the pump could still be writing an event or a heartbeat to the response. net/http does not allow a write to a response after ServeHTTP has returned. The handler now waits for the pump to exit; a pump blocked in a write returns once the connection is gone. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
86da6c7a5b |
style(lineage-webhook): prefix sentinel error names with Err
Go names sentinel error variables Err...; the package is internal and the webhook is their only user. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
648928f272 |
refactor(workloadmonitor): use k8s.io/utils/ptr instead of pointer
The pointer package is deprecated; its Bool and BoolPtr are aliases of ptr.To, so the values written are unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
939a1ec44b |
refactor: read managed fields through FieldsV1 accessors
Direct access to FieldsV1.Raw is deprecated; GetRawBytes returns the same bytes and is nil-safe, and NewFieldsV1 builds the test fixtures. The write-options test still covers a raw-only orphanDependents delete option, now decoded from its wire form, and asserts that the converted options keep it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
7a481c4765 |
chore(workloadmonitor): discard best-effort close errors explicitly
The Prometheus response body is closed after it has been read, and the test stubs' writes fail on the client side of the test, so neither error changes a result. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
87b794a992 |
chore: handle errors dropped in cleanup paths and tests
The drain tests now assert that RunThenDrain returns the run's nil error, the idle-tail test fails on any error other than the stream ending or being cut, and a test fixture file fails the test when it cannot be closed. Removing a temporary directory and closing a response body that has been read are best-effort: a failure leaves nothing for the caller to act on and does not change any result, so those errors are discarded explicitly. The same holds for the stub servers' response writes, whose failure surfaces on the client side of the test. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
bd998d9653 |
fix(kube-ovn): fail the plunger when its status server cannot start
The status stream was started in a bare goroutine whose error was dropped, and the runnable's registration error was discarded too, so a port that could not be bound left the plunger running with no status endpoint and nothing in its logs. The server now runs inside its runnable: a listen error fails the manager, as a metrics server bind failure does, and stopping the manager is not reported as an error. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
41ad82842a |
fix(sse): drop a client as soon as a write to it fails
Write errors on the event stream were ignored, so a client whose connection broke kept its handler, pump and registration until the request context ended, and every event was written into the dead connection meanwhile. A failed write now ends the stream and unregisters the client, and the handler also returns once its pump ends, so Shutdown no longer leaves streams open. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
20ce6c691e |
fix(sse): close a client's channel once when the server shuts down
Shutdown closed every open client's channel, and the client's handler closed the same channel again when its request ended, so any stream still open at shutdown panicked with a close of a closed channel. Stopping a client is now idempotent. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
473cfdf478 |
Merge branch 'main' into feat/observability-proxy-exported-namespace
Assisted-by: LLM Signed-off-by: Maxim Belyy <maximbel2003@gmail.com> |
||
|
|
a4ea942a2c |
Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM Signed-off-by: Maxim Belyy <maximbel2003@gmail.com> |
||
|
|
34843c1a91 |
Merge branch 'main' into feat/observability-proxy-metrics
Assisted-by: LLM Signed-off-by: Maxim Belyy <maximbel2003@gmail.com> |
||
|
|
906eeb2372 |
feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does
Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.
None of the existing strategies fits a tenant cluster:
- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.
**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):
1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.
**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:
- **Application gone**:
1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.
Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.
Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:
- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.
Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.
A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.
Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.
### Verification
- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
- the full backup path;
- in-place restores with default options and etcd only;
- a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
- RestoreJob deletion resuming a stopped cluster;
- the stop deadline;
- the preflight: missing or incomplete Velero backup, missing dump;
- a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.
Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
- a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
- keys are moved from the old prefix to the new one;
- an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.
### Not in this PR
- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.
### Screenshots
N/A — no UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.
### Release note
```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
04fa87ccd1 |
style(backups): clear staticcheck findings in overlapping controllers
Drop a field-manager constant nothing uses in the MariaDB driver, switch on the MongoDB operator's state as a tagged switch, and reword the error strings of the MongoDB driver and the workload monitor so they do not start with a capital. The errors keep their meaning and nothing matches their text. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4c3a4b42cc |
style: clear unused code and style findings reported by staticcheck
Remove helpers and constants nothing references, escape format characters in test literals, and take the simplifications staticcheck suggests where they read the same. Error strings no longer start with a capital: they keep their meaning, with the kind or component name moved out of first position. Callers match errors by type, not text; the one test that matches text, for Kafka, keeps the phrase it looks for. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e978ed7f2f |
fix(postgres): stop backup jobs from fighting over the WAL archive destination (#5221)
## What this PR does Part of #5202. A second PR will restore the last good destination when a job that changed it fails. The ObjectStore a Postgres Cluster archives its WAL to is shared by every backup of the application, yet every running BackupJob server-side applied its own strategy's destination onto it every five seconds with forced ownership. Two BackupJobs of one application with different BackupClasses flipped the destination back and forth until the healthy backup failed too, and a webhook or apiserver blip during that apply failed the BackupJob at once. - A BackupJob writes the destination and the Cluster plugin only before its `cnpg.io/Backup` exists. Afterwards it only reads the live Cluster, and a failed read is returned for a controller retry instead of failing the running backup. - Before writing, a BackupJob waits, with a `Ready=False` / `WaitingForOtherBackup` condition naming the other job, while another running BackupJob of the application has a `cnpg.io/Backup` in flight. The wait is not capped, for the same reason as in #5211. The queue reads the Cluster's own Backups from the apiserver; with leader election one reconcile runs at a time, so only a leader handover can still let two jobs through. An atomic per-application claim is #4559. - Before writing, a BackupJob also checks the apiserver for a restore of its application and holds while one runs, including a job released from the queue: a write landing during the restore would put the restored Cluster back on the source's WAL prefix. - Write errors that cannot succeed on retry (invalid, bad request, forbidden) still fail the job at once. Any other is retried with a `ClusterPluginApplyRetrying` condition for the backup deadline, counted from the first failure since the job last waited, and recorded in a new `DestinationApplied` condition. - The pass that creates the `cnpg.io/Backup` records on it the destination, endpoint, credentials Secret and serverName it wrote (`backups.cozystack.io/cnpg-written-destination`), and the artifact and its size probe are built from that record, so a strategy edited, or a Cluster re-rendered, while the backup runs does not redirect the artifact. A Backup without the record falls back to the strategy as before. - The plugin rollout deadline and the instance check count from the write that attached the plugin, not from the job's start. The apply that attaches it also stamps the time on the Cluster (`backups.cozystack.io/barman-plugin-attached-at`), so a lost BackupJob status write after the attach does not fall back to an older time, and a job that starts while another job's recent attach is still rolling waits for that rollout too. A stamp a few minutes ahead of the controller's clock is waited out; one further ahead is treated as corrupt. Tests drive the reconciler through a running backup that must not rewrite the destination, the queue and its release, a restore starting while a job waits, each class of write error, a recovered write, a failed Cluster read while the Backup runs, lost status writes around the attach, another job's rollout, clock drift on the attach stamp, the recorded destination and serverName, and the rollout deadline. Each guard was checked by mutation. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(postgres): Postgres BackupJobs of the same application no longer overwrite each other's WAL archive destination: a job writes it only before its backup starts and waits while another backup or a restore of the application is in flight. A transient error while writing it is retried instead of failing the job at once, and the backup artifact names the destination and serverName the backup was actually written to. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Reliability** * CNPG backups wait while another backup or restore is using the shared destination, and retry temporary destination-write failures while observing the backup deadline. * Backup artifacts and size checks use the destination recorded when the backup was created, even if destination settings later change. * Backup processing handles plugin rollout timing and destination-hold release more consistently. * **Restore** * Restore operations account for shared WAL archive destination conflicts and guard against incompatible backup selections. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
583f18437d |
test(clickhouse): give the credentials-wait BackupJob a UID
The test pins that a backup held for late bucket credentials keeps its archive name, and a run's archive name now carries part of the BackupJob's UID, so the fixture needs one for the name it expects to be the name the controller mints. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dae76d186c |
Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for late bucket credentials, against the controller's archive helper. That helper now names a run's archive with part of the BackupJob's UID, so the merged test has to give its BackupJob one. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5c6fc10a23 |
test(postgres): pin three guards no test caught when changed
A job whose own Backup exists skips the wait for another job's plugin rollout, and the size probe keeps the strategy's Secret key names while it takes the Secret from the record, adding a credentials block only when the strategy has none. Each of these could be changed with the suite still green: dropping the own-Backup check, or forcing the missing-credentials branch either way. Cases for a job with a running Backup next to a recent foreign attach, a strategy without credentials and a strategy with custom key names now fail on each change. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b3ca798bc4 |
fix(backups): read the runs that may own a MariaDB dump uncached
Before deleting a dump the cleanup checks every other operator Backup on the same prefix that could have written it, and it read them from the informer cache. A run whose operator Backup had not reached the cache yet but had already uploaded its dump was missed, and its dump deleted as this Backup's. Both the recorded and the legacy paths now list the runs from the API server, as the restore checks already do. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
238b3ee491 |
fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but only the `s3_lifecycle` job of its plugin worker applies it, and that job is scheduled by the admin server. Cozystack deployed neither, so a lifecycle rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike. That also made the documented way to bound backup storage a no-op. This PR adds the admin server and a worker to Simple and MultiZone SeaweedFS instances, behind a new `lifecycle.enabled` value that is **off by default**. A Client instance is unchanged, because its rules are enforced where the instance it connects to runs. The worker registers for `s3_lifecycle` only, so vacuum, balancing and erasure coding are not started as a side effect. The admin keeps its job state on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default when that is empty; `volume.storageClass` is not reused, because in MultiZone it can name a class local to one zone or node. The admin UI is protected by a generated password, because a tenant namespace admits traffic from the whole cluster. The dashboard Role lists the two new WorkloadMonitors. Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart renders no `[grpc.s3]` section, so the s3 gateway served its internal gRPC in plaintext while the worker dials it with mTLS, and every lifecycle delete failed the TLS handshake. The section is backported from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart patch, applied with `--forward` so the next `make update` stops at the point where the patch must be dropped instead of silently reversing it. The s3 pods are restarted once so they load the section. This also repairs IAM cache propagation between s3 replicas, which failed in the same way, and it applies whether or not lifecycle is enabled. The data-plane release now retries a failed install or upgrade in place (`RetryOnFailure`) instead of rolling it back. With the previous default remediation and unlimited retries, an upgrade that waited out its timeout, for example on a tenant quota with no room for the admin volume, rolled back and retried forever, restarting the s3 pods twice per cycle. The worker's hostname anti-affinity is relaxed to a preference so a rollout finishes on a single-node cluster. #### Why lifecycle is off by default The daily walk that SeaweedFS uses to expire objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when no identity is given; it does not re-check the object's age either. A key overwritten between the walk's listing and the delete loses its new content. This path is unchanged from 4.45 through 4.52 and on upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`, `identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup buckets write unique keys and are barely exposed; a tenant bucket that rewrites a fixed key is. On top of that, turning enforcement on would start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so nothing changes for existing instances until `lifecycle.enabled` is set. Making it the default again is left to a separate PR that carries object identity on the walk path; upstream report: https://github.com/seaweedfs/seaweedfs/issues/12451. #### Turning lifecycle on and off When `lifecycle.enabled` is set, rules that have been stored but ignored start applying within minutes, and objects they have already expired are deleted on that first pass, so review existing bucket lifecycle rules first. Set `lifecycle.storageClass` before enabling, and prefer a replicated class: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. Moving the volume means disabling lifecycle, deleting the `admin-data-seaweedfs-admin-0` PVC, changing the class and enabling lifecycle again. Each instance gains two small pods and a 1Gi PVC; when the tenant quota has no room for them, the data-plane release keeps retrying in place until it has. When `lifecycle.enabled` is turned off again, the parent release removes the admin credentials Secret while the data-plane release removes the admin StatefulSet only on its own reconcile, shortly after. An admin pod that restarts in that window fails to start on the missing Secret until the StatefulSet is gone; nothing is lost, and the order is left as is. The admin PVC stays until the instance is deleted, and a later re-enable starts the admin with a new password. Verified on a test cluster with the 4.45 chart: without the backport, the worker found the expired objects and every delete failed with `tls: first record does not look like a TLS handshake`; with this branch and lifecycle enabled, a rule with a past expiration date removed the matching objects and left the others, both on a manual run and on the first scheduled pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up needed, not opened yet: `operations/services/backup-classes.md` (v1.7 and next) advises an S3 lifecycle policy, which the bundled SeaweedFS applies only once `lifecycle.enabled` is set; the SeaweedFS reference page gains `lifecycle.enabled` and `lifecycle.storageClass` from the regenerated README on the next release - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The terraform provider does not model the SeaweedFS application, so the new values do not reach it. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added optional, daily S3 bucket lifecycle-rule processing for supported SeaweedFS configurations. It is disabled by default; Client instances rely on lifecycle rules configured on the connected SeaweedFS. * Added configurable storage for lifecycle job state and dashboard visibility for lifecycle components. * Enabled the S3 gRPC server to use its internal TLS identity when security is configured. * **Bug Fixes** * Improved storage-class handling so numeric-looking names remain strings and unset classes are omitted where appropriate. * Clarified backup cleanup guidance, including when lifecycle rules apply and how to reclaim storage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3badbdef74 |
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
1d22013646 |
fix(postgres): hold a backup until the plugin sees its new destination
The plugin-barman-cloud sidecar reads the ObjectStore through its own cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud v0.13.0) and invalidated only by the sidecar's own writes, and WAL archiving keeps it warm. The pass that changed the destination created the cnpg.io/Backup a moment later, so the base backup could be taken against the previous destination while the artifact names the new one, and a restore from it finds nothing. On a live cluster with the shipped plugin, a Backup created two to four seconds after switching the destination landed in the old bucket in two of three tries; with a fifteen second wait it landed in the new one every time. The apply that changes the ObjectStore configuration now stamps the time on the ObjectStore itself, so the record cannot be lost apart from the change, and the job creates its Backup only once that change is older than the cache can be. A change is judged on the configuration a server-side dry run of the same apply returns, so values the apiserver defaults or another field manager owns do not read as a change on every pass. That wait leaves a job with the destination written and no Backup yet, which the queue, looking only for Backups in flight, did not see: a second job with another destination wrote its own, the first took that as a change and stamped again, and the two restarted each other's wait forever. Every apply of the ObjectStore therefore also names the job that made it, which only a job without a Backup of its own and let through by the queue does, and the queue holds every other job while that one runs and has no Backup yet. Once it has one, that Backup holds the destination for as long as CloudNativePG runs it, as before, so a job stuck after its Backup finished does not hold every later backup of the application. This also covers a job waiting for the rollout of the plugin it attached, during which a second job used to switch the destination as well, including when its own write left the configuration as a restore had re-rendered it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5a95bcdd64 |
fix(clickhouse): keep a run's archive name within the old name's length
The archive name is part of every object key, a key is capped at 1024 bytes, and the legacy s3PathOverride prefix has no length limit. The full UID made the name 37 characters longer than before, so a prefix that fit the old name pushed the manifest key past the cap and failed the backup. The run's name now drops "clickhouse-" and the "T" and "Z" of the timestamp and spends the room on the first 12 hex digits of the UID, so it is exactly as long as the old name for every application. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9978771858 |
feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does Adds `observability-proxy`, a new system component that lets a tenant user read the logs of their own namespace from VictoriaLogs with the same Kubernetes credentials they already use for the console, and with Kubernetes RBAC deciding what they can read. This is the first part of the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94); this PR implements the namespace logs root only. Metrics and the per-application roots follow in separate PRs. Why a proxy: today the log stores are shared per root tenant and have no notion of a caller, so tenants cannot be given direct read access without seeing other tenants' logs. The proxy authenticates the caller, authorizes the request from its path, and rebuilds the upstream query so that its scope cannot be widened by the client. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. What is included: - `cmd/observability-proxy` and `internal/observabilityproxy`: the proxy. - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>` for the VictoriaLogs read endpoints the console needs (`query`, `hits`, `stats_query_range`, `tail`, `field_values`, `stream_field_values`, `facets`). Anything else under the root answers `404`. - Authentication by `TokenReview`, authorization by one `SubjectAccessReview` built from the path (`list` on `cozystack.io/logs` in the namespace), both cached briefly. - The upstream request is rebuilt from an allowlist of parameters; the namespace scope is always forced as `extra_stream_filters`. Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL `options(...)` block, and tenant headers such as `AccountID`/`ProjectID` are not forwarded or are refused. - Non-canonical paths (`..`, `//`, encoded separators) are refused with `400` before any authorization is built, because the console nginx matches on the normalized path but forwards the raw one. - Time window, step, bucket count and result limits are clamped; per-user rate, concurrency and open-tail limits answer `429`. Invalid time, step and limit parameters answer `400`. - The store is resolved from the namespace's `namespace.cozystack.io/monitoring` label. - Live `tail` streams are passed through unbuffered, and the upstream tail request disables compression so headers reach the client before the first line. - `packages/system/observability-proxy`: chart and image build, wired into the root `make build`. - Platform wiring: a new `cozystack.observability-proxy` PackageSource, enabled by default on the full variants and opt-in on the slim ones, the same way as `monitoring-agents`. - RBAC: an aggregated `cozy:observability:view` ClusterRole grants `list` on `cozystack.io/logs` to tenant `view` and above. No API server serves this resource; the rule exists only for the proxy's `SubjectAccessReview`. The proxy's own ClusterRole is limited to the calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch` on namespaces). - Console nginx: when the proxy is not installed or cannot be reached, the observability roots answer `404` with a Kubernetes `Status` body, which the console shows as not available on this cluster; the proxy's own `502`/`504` for an unreachable store pass through unchanged. - Console: one regex `location` in the console nginx routes the four observability roots to the proxy ahead of `/apis`. The upstream is resolved at request time, so the console starts even when the proxy is not installed; those paths then answer `404` with a Kubernetes `Status` body. The console Deployment now receives the cluster domain for that resolution. - E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/` with two tenants, covering `401`/`403`/`200`, scope-widening attempts, path normalization and a long-lived `tail`. #### How it was tested - `go test -race` and `go vet` for the proxy packages. - helm-unittest for `observability-proxy`, `dashboard`, `tenant` and `core/platform`; bats tests for the console nginx routing. - The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs as the root store and refuses to run when a real one exists. Its cases were replayed by a script against the real store on a test cluster. - Verified on a test cluster: - `401` without a token and with an invalid token, `403` for another tenant's namespace, `200` for the caller's own. - Client `extra_stream_filters`/`extra_filters` (GET and POST), an `AccountID` header and another tenant's stream named inside the query do not widen the scope; `options(...)` answers `400`; a path with `..` pointing at another tenant answers `400`; an endpoint outside the allowlist answers `404`. - Through the full browser chain (OIDC login, gatekeeper, gateway, console nginx), a single `tail` stayed open for more than two minutes without reconnecting, and requests from the same session to other tenants' roots answered `403`. - Not verified: the `429` limits under real load. `golangci-lint` has not been run locally. #### Known limitations - The proxy is reachable only through the console route; tenant workloads have no egress rule to it. Every request takes its concurrency or tail slot before the body is read; the token and access reviews and the body read run under `--request-timeout`, so a stalled kube-apiserver answers `503` rather than holding the request; and writes to the client carry a deadline (renewed per line for tails). `--request-timeout=0` disables those bounds and should not be used in production. - The first lines a newly started pod writes can be missing from an already open `tail`, although they are in the store. The likely cause is that Fluent Bit picks up a new log file later than the VictoriaLogs live-tail window; later lines of the same pod arrive. A history query shows them. - The per-application roots (`.../<plural>/<name>/log`, `.../metrics`) and the namespace metrics root answer `404` in this PR. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/787 (docs land once the implementation PRs merge) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: `cozystack/website` is reached. A new platform component (`packages/core/platform/sources/observability-proxy.yaml`) triggers `content/en/docs/next/guides/platform-stack/_index.md` and `operations/configuration/licenses.md`; adding it to the system bundle (`packages/core/platform/templates/bundles/system.yaml`) may also need a line in `operations/configuration/variants.md`. A user-facing page for the logs API and the new tenant `view` permission is not required by the trigger map but would be useful. Tracked in cozystack/website#787. Other repositories: no trigger matched. `packages/core/platform/values.yaml` is not changed (only a test fixture), no package under `packages/apps/` gains a values field, and no CRD changes. The root `Makefile` `build` target builds one more image; this does not touch `hack/` or the make targets `ccp` relies on, so `ccp` is not considered affected, but a reviewer may want to confirm. ### Release note ```release-note feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added an observability proxy for tenant-scoped log access through the console, including live log streaming. Requests are authenticated, access-controlled, and subject to configurable rate and concurrency limits. * Added platform packaging and deployment support, with console routing for observability requests. * **Tests** * Added automated coverage for access control, tenant scoping, query handling, streaming, proxy behavior, and deployment configuration. * **Documentation** * Updated end-to-end test guidance with host requirements for running the observability proxy suite. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a6a704768c |
test(backups): pin a failed RestoreJob listing in the MariaDB cleanup
A RestoreJob listing that fails says nothing about whether a restore reads the dump. Swallowing that error left the whole suite green, while it would let the cleanup delete the dump under a running restore. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f2507b4b84 |
docs(backups): name what the MariaDB cleanup reads uncached
The reader's comment named only RestoreJobs, and the restore path said the cleanup holds a Backup only for RestoreJobs it can see, while the cleanup now lists both RestoreJobs and operator Restores straight from the API server. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0195a66fae |
fix(backups): release a MariaDB Backup when the operator CRDs are gone
Listing the operator Restores of a Backup before reading its operator Backup made the cleanup fail on a cluster without the mariadb-operator CRDs, retried for good, so the Backup and its namespace never went away. The operator Backup read that follows already releases the Backup in that case; with no Restore kind no restore can run, so the listing now lets the cleanup reach it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
8246808289 |
fix(backups): hold a MariaDB dump while an operator Restore reads it
The cleanup waited for a restore by the phase of its RestoreJob, but the phase is not the restore: a RestoreJob fails on its deadline while the operator Restore it created keeps running, and that Restore still reads the dump. Deleting the dump then made mariadb-operator restore the nearest other one. The cleanup now also holds the Backup while an operator Restore of its operator Backup has not been marked Complete, which mariadb-operator does once the Restore's Job has completed or failed, so what keeps the dump is the restore that reads it rather than the status of the request. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ee6404a0c8 |
fix(backups): retry a failed lookup of a MariaDB operator Restore
The lookup now goes to the API server, which can time out or throttle, and a failed lookup failed the RestoreJob. It says nothing about whether the operator Restore exists, so the RestoreJob could be failed while its Restore was running, and the Backup's cleanup would then stop waiting for it and delete the dump the Restore reads. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
14a842ad16 |
fix(backups): find a MariaDB operator Restore the cache has not seen
The MariaDB restore creates its operator Restore once and then follows it, finding it by label in the informer cache. Nothing watches operator Restores, so a reconcile that ran before the new Restore reached the cache saw none: it created a second one, or, now that a restore refuses a Backup being deleted, failed the RestoreJob while its Restore was still running, after which the cleanup deleted the dump that Restore was reading. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d57ebbe808 |
docs(backups): state what the MariaDB dump delete does not cover
The comment on who deletes a MariaDB Backup listed namespace teardown without saying that teardown can remove the operator Backup first, in which case the dump is left behind with a Warning, so it promised a delete the code does not make. The remover contract claimed a missing key is never an error, which holds only for a backend that answers the delete with 204, the one status minio-go accepts. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
3334152da7 |
fix(backups): report a recorded MariaDB dump that was not found
A recorded dump missing from the listing was logged as deleted, with no event. That is right when an earlier attempt or maxRetention removed it, but the listing goes through the platform's current endpoint, and after an endpoint move the dump can still sit on the old storage: the cleanup then deleted the operator Backup that pointed at it and said it had deleted the dump. The cleanup now records a Normal ArtifactAlreadyAbsent event naming the key and the endpoint it listed, so whoever later looks for that dump can tell where it was not found. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
bbeca34535 |
fix(backups): retry a failed read of the platform backup credentials
The MariaDB cleanup reaches the platform bucket through the platform's current endpoint so that a Backup taken before the endpoint moved is still deleted. A failed read of the platform credentials made it fall back to the endpoint recorded on the operator Backup, which may be the old address: there the dump is not found, a missing dump counts as already deleted, and the operator Backup that pointed at the real dump was deleted with a success log. A read that fails with an API error now holds the Backup and retries. A missing or malformed platform Secret still falls back, since it is a standing state that a retry would not change. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
37a74e5939 |
test(backups): reach the S3 calls the MariaDB cleanup timeout bounds
Contesting a recorded dump lists the bucket before the delete, and the timeout test supplied no lister, so a real client failed on DNS before the blocking delete was ever called and the suite stayed green with the timeout removed. The test now lists the dump through the fake bucket, blocks in the listing and in the delete in turn, and requires that the blocking call was entered. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
654c543264 |
fix(backups): see a MariaDB restore the cache has not caught up with
A MariaDB restore now checks on the API server that its Backup is not being deleted before it starts, so a restore that started first exists on the API server before the delete. The cleanup read RestoreJobs from the informer cache, which can lag behind, and could delete the dump under a restore it had not seen yet. Reading them uncached closes that window from both sides. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
320d0299ad |
fix(backups): refuse a MariaDB restore from a Backup being deleted
Deleting a MariaDB Backup now deletes its dump, and mariadb-operator restores the dump nearest the requested time when the exact one is gone. A restore that starts while the Backup's cleanup runs can lose its dump between the check and the operator's read, and then completes on another run's data without any error. The restore now reads the Backup from the API server before creating the operator Restore and fails when it is being deleted or is gone. The copy the reconcile was handed comes from the cache and can predate the delete; a transient read error keeps the RestoreJob waiting like any other failed dump check. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |