mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
a4ea942a2ca2e406d0be98bc730d4f87ce570fe9
4788
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a4ea942a2c |
Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM Signed-off-by: Maxim Belyy <maximbel2003@gmail.com> |
||
|
|
b6f5a4fa35 |
feat(monitoring): stamp application identity on scraped series (#5216)
## What this PR does
Adds a default scrape class `application-identity` to both VMAgents: the
cluster-wide one in `monitoring-agents` and the per-tenant one in
`monitoring`. Every series scraped from a pod that carries the lineage
labels `apps.cozystack.io/application.{group,kind,name}` gets them as
`apps_cozystack_io_application_{group,kind,name}`.
Why: the per-application metrics scope in the design proposal [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94) (Design 4a)
needs to tell which application an exporter series belongs to. Today
that is only possible indirectly, through pod names.
How the labels are protected against forgery by an exporter:
- At the target stage, the class always sets all three labels: to the
pod's real lineage value, or to a placeholder `-` when the pod has none
(`-` is not a valid Kubernetes label value). With `honorLabels: false`,
any copy the exporter publishes therefore collides with the target label
and is renamed to `exported_*`.
- At the metric stage, the class drops every `exported_*` copy.
- The placeholder is removed by the VMAgent's global remote-write
relabeling (`inlineRelabelConfig`), not at the metric stage: vmagent
adds `up` and `scrape_*` with the target labels after metric relabeling,
so only the remote-write stage sees those series. Targets without
lineage therefore keep exactly the label set they have today on every
series, including `up` and `scrape_*`.
- With `honorLabels: true` the exporter's value wins and no relabel rule
can tell it apart. ingress-nginx scrapes with `honorLabels: true`, and
its series describe Ingress objects in any namespace, so the
controller's identity would misattribute traffic; its scrape now drops
the identity labels explicitly, from the target (so `up` and `scrape_*`
stay unchanged) and from the scraped samples (so an exporter's copy
cannot survive). Switching it to `honorLabels: false` was not done,
because that renames `namespace` and breaks the existing ingress
dashboards.
#### How it was tested
- helm-unittest for `monitoring-agents`, `monitoring` and `ingress`.
These tests check the rendered config only; the relabeling behaviour was
verified offline by running vmagent v1.136.0, built from source, against
the rendered rules: `up` and `scrape_*` carry no identity for targets
without lineage, and the placeholder appears on no series.
- The rendered `scrapeClasses` validate against the vendored VMAgent CRD
schema.
- The rendered rules were exercised on a locally built vmagent v1.136.0
(the version the bundled operator deploys) writing to a local
VictoriaMetrics: with and without lineage, with and without a forging
exporter, `honorLabels` true and false, and the ingress-nginx endpoint
rules. Checked on exporter samples and on `up`, `scrape_samples_scraped`
and `scrape_series_added`; the previous rules reproduce the `-`
placeholder on the generated series of targets without lineage, the new
ones leave no placeholder on any series.
- Verified on a test cluster with the earlier revision that removed the
placeholder at the metric stage (the move to remote-write relabeling is
not yet verified on a cluster): series carry the application labels;
metrics read for a Redis instance through the per-application proxy root
contained only that instance's pods; vmagent logs show no errors.
#### Known limitations
- Scrapes with `honorLabels: true` other than ingress-nginx are not
protected against an exporter that publishes the identity labels itself.
Today these are the vendored qdrant and harbor ServiceMonitors; their
exporters are platform images with fixed label names, and their series
describe the application's own pods.
- A tenant's raw `vmagent.inlineScrapeConfig` bypasses scrape classes
entirely, and so do `additionalScrapeConfigs`. VMProbe takes only auth
settings from a scrape class (operator v0.68.4); no VMProbe exists in
the platform today.
- A scrape object's own relabel rules run after the class and can
overwrite the labels; tenants cannot create VictoriaMetrics operator
objects, so this is platform-controlled.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff changes templates and tests in `packages/system/monitoring-agents`,
`packages/system/monitoring` and `packages/extra/ingress` without
adding, renaming or removing a package and without touching any
`values.yaml` or schema. The telemetry metrics the telemetry server
reads (`cozy_cluster_info`, `cozy_application_count`) are produced by
the operator, not by these scrapes. The proposal text in
`cozystack/community` (#94, Design 4a) should be corrected to name
qdrant and harbor as further `honorLabels: true` scrapes; that is not a
trigger-map item.
### Release note
```release-note
feat(monitoring): VMAgent now adds `apps_cozystack_io_application_{group,kind,name}` labels to every series scraped from a pod that carries application lineage labels. Because the label set of those series changes, each of them becomes a new series after the upgrade: graphs and dashboards show a break at the rollout, queries over a range that spans it see two series per pod, and alert `for:` timers on those series restart. Series of targets without lineage labels, including node-level series (cAdvisor, kubelet) and kube-state-metrics, and all ingress-nginx series keep their current label set; this includes their `up` and `scrape_*` series.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Improvements**
* Monitoring metrics now use application identity labels derived from
Kubernetes pod metadata for covered scrape targets. Exporter-supplied
duplicates and placeholder values are removed, including from `up` and
`scrape_*` metrics.
* Ingress metrics no longer retain application identity labels at either
target or metric relabeling stages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
906eeb2372 |
feat(backups): add Kubernetes backup strategy for tenant clusters (#4931)
## What this PR does
Adds a `Kubernetes` backup strategy so a tenant Kubernetes application
can be backed up and restored through the backups framework, and routes
`apps.cozystack.io/Kubernetes` to it in `cozy-default`.
None of the existing strategies fits a tenant cluster:
- Its API state is a key prefix (`/<namespace>_<release>/`, the
TenantControlPlane's datastore schema) in the etcd `DataStore` shared by
every cluster of the tenant. The `Etcd` strategy snapshots the whole
DataStore, so restoring it would roll every tenant cluster back at once.
- Its control-plane CA, front-proxy CA, service-account key and Talos
secrets cannot be regenerated: new ones break every node, kubeconfig and
token issued under the old ones.
- Its PersistentVolumes are DataVolumes in the tenant namespace
(`pvc-<tenant pvc uid>`, labelled `cluster.x-k8s.io/cluster-name`), and
deleting the application deletes them.
**Backup** (`strategy.backups.cozystack.io/Kubernetes`, driver in
`internal/backupcontroller/kubernetesstrategy_*.go`):
1. Checks the control-plane and Talos certificates and sets a
`CertificatesExpiring` condition (plus a Warning event) when a CA
expires before the backup's TTL or a leaf within
`certificateWarningThreshold` (720h). This never fails the run.
2. A Job in the application namespace dumps the cluster's prefix with
`etcdctl` to the platform bucket. It authenticates with the cluster's
own `kubernetes-<name>-datastore-certificate`, the one its
kube-apiserver uses. It runs on a new `kubernetes-backup` image (alpine
+ curl + jq + etcdctl matching the platform's etcd).
3. A Velero backup takes the PKI Secrets (selected by
`kamaji.clastix.io/name` / `app.kubernetes.io/instance`; the driver adds
the instance label to the cert-manager-issued Talos CA Secrets, which
carry neither) and the data DataVolumes/PVCs. Worker system disks are
not backed up.
4. The Backup records the prefix, the endpoints, the data DataVolumes
and the HelmReleases of the cluster and its `KubernetesNodes` pools. The
HelmReleases are not in Velero because the lineage webhook labels worker
disks after the same applications, so a label selector for the releases
would also pick the disks.
**Restore** goes into the application the backup came from (rename /
cross-namespace is rejected). `RestoreJob.spec.options` takes `etcd` and
`volumes` (both default `true`) to restore either half alone:
- **Application gone**:
1. Restore the PKI Secrets and the volumes.
2. Recreate the cluster's HelmRelease. The Kamaji release this repo
ships gives a new TenantControlPlane a UID-keyed datastore schema, so
the restore waits for that schema.
3. Stop the new control plane, write the dump under the new prefix (keys
are moved from the old prefix), and start it again.
4. Recreate the worker pools last, so workers join the restored cluster
with the restored CA and Talos secrets rather than an empty one.
- **Application live**: suspend the cluster's HelmReleases, then stop
the MachineDeployments and the cluster-autoscaler (for `volumes`,
deleting the backed-up DataVolumes/PVCs once the Machines are gone), and
then the KamajiControlPlane (for `etcd`). Workers go first because CAPI
removes a worker's Node through the tenant API. Then write the selected
halves back and restore every replica count. Each original count is
recorded on the object it belongs to, so the cluster is put back on
success, on failure, after a 30-minute stop deadline, and when the
RestoreJob is deleted midway. A retry keeps the counts from before the
first attempt, and a HelmRelease suspended before the restore stays
suspended.
Before changing anything, a restore checks that the Velero backup is
still there and Completed, and that the dump object exists (a
verify-mode run of the Job). The restore mode is recorded in the
RestoreJob's status, so a RestoreJob cannot claim that a live cluster is
gone.
Before it runs `del --prefix`, the restore Job downloads and validates
the dump and saves the current prefix to `<dump>.pre-restore`. That copy
is never overwritten, so a retry cannot replace it with a partial
prefix, and it is dropped after a successful restore. The Job writes a
marker to its termination message, and the outcome depends on it:
- A failure before `del` resumes the cluster.
- A failure after `del` leaves the control plane stopped
(`KubernetesEtcdWriteFailed`) rather than started on a partial prefix.
- Deleting the RestoreJob while its restore Job runs also leaves it
stopped.
Events and master leases are skipped only as top-level resources. Velero
cannot overwrite an existing claim, so a live-cluster volume restore
deletes the current claims first; this is documented.
A backup whose Velero half fails deletes the etcd dump it already
uploaded, and a transient error creating the Backup is retried rather
than failing the run. Deleting a Backup removes the Velero backup
(DeleteBackupRequest) and the dump object plus its pre-restore copy (a
cleanup-mode Job, given up after five failed attempts with a Warning,
plus the same skip-artifact-cleanup escape hatch as Redis/Rabbitmq).
With `velero.bslEnabled=false` the strategy is not rendered, and the
DefaultObjectsGate does not count it as missing, the same as the Velero
strategies. Like every default strategy, it is also gated on its CRD
being established, so the upgrade that first ships the CRD to an
installed chart cannot wedge the release.
Also included: RBAC for
TenantControlPlanes/DataStores/KamajiControlPlanes/MachineDeployments/Machines/Deployments,
chart values (`kubernetesBackupClientImage`,
`kubernetesBackupActiveDeadlineSeconds`, `kubernetesBackupResources`), a
`make image-kubernetes-backup` target, docs in
`docs/operations/backup-classes.md`, and the
`examples/backups/kubernetes/` walkthrough.
### Verification
- `go test ./internal/backupcontroller/`: the new tests cover:
- the certificate windows, pool HelmRelease discovery, option parsing
and the restore-phase selectors;
- the full backup path;
- in-place restores with default options and etcd only;
- a pre-suspended HelmRelease;
- a failed etcd write leaving the control plane stopped, including on
RestoreJob deletion;
- RestoreJob deletion resuming a stopped cluster;
- the stop deadline;
- the preflight: missing or incomplete Velero backup, missing dump;
- a rebuild, and an unreadable HelmRelease record;
- rename rejection, a non-etcd datastore, and the cleanup attempt limit.
Each safeguard in the Go driver was checked by mutating it out and
confirming that a test fails. The rendered Job script has no committed
test: a unit test would have to extract the script from the rendered
chart, and its behaviour belongs in the chainsaw e2e follow-up instead.
It was run against stub `etcdctl`/`curl` in every mode and branch:
- a foreign key or an empty value stops it before any etcd call;
- an S3 answer other than 200/404 on the pre-restore copy stops it
before any etcd call;
- an existing copy is kept, and a missing one is uploaded before `del`;
- keys are moved from the old prefix to the new one;
- an object in a namespace named `events` is kept.
- `helm unittest` (29/29), `hack/backupstrategy-crd-gate.bats` and `helm
lint` on `backupstrategy-controller`.
- On a live cluster (Velero 1.18.1), with a tenant cluster holding a
ConfigMap marker and a file on a PVC:
- The same flow done by hand before the driver existed: back up, delete
the application, rebuild. The CA, front-proxy CA, SA key and Talos
secrets were byte-identical afterwards, the pre-backup admin kubeconfig
still worked, namespace and ConfigMap UIDs were unchanged, and the PVC
data came back.
- Through this driver: the BackupJob succeeded (432 keys dumped, Velero
backup Completed, Backup metadata as described above). The tenant
namespace holding the markers was then deleted, which also deleted its
PVC and the backing DataVolume, and a RestoreJob with default options
rolled the live cluster back. It stopped the workers and then the
control plane, wrote the prefix back, restored the volume, and restored
the control plane to 2 replicas, the pool and the autoscaler to 1, and
unsuspended the HelmRelease. A new worker joined. The namespace came
back with its original UID, the ConfigMap and the PVC (bound to the same
volume) with their data. That run caught an ordering bug, now fixed and
covered by a test: scaling the control plane down first left the old
Machine stuck in `Deleting`, because CAPI removes the Node through the
tenant API.
- On the current head, through the driver, against the same tenant
cluster:
- **Backup.** It succeeded. The volume list it recorded was taken before
Velero started.
- **`{"volumes":false}` restore after a ConfigMap was deleted and
another created.** The deleted ConfigMap came back and the newer one was
gone. The worker was left running and the volume data was untouched. The
restore waited for terminating control-plane Pods before writing. The
pre-restore copy was saved before `del` and dropped afterwards.
- **Rebuild after the application was deleted, with default options.**
It ran the full sequence: secrets, volumes, cluster HelmRelease,
datastore schema, stop, etcd write, resume. It succeeded in about 3
minutes. The CA and Talos CA were byte-identical, a new worker joined,
and the namespace and ConfigMap UIDs and the PVC data were back. The
tenant cluster runs on a Kamaji that still names schemas `<ns>_<name>`,
so this run did not exercise a prefix change; that path is covered by
the unit tests and the stub runs.
### Not in this PR
- A chainsaw e2e. The tenant Kubernetes suites are bounded by tight op
budgets that tests pin, so the round trip needs its own suite or a
budget change; it will be a follow-up.
- `rotationPolicy` on the Talos CA Certificate in the kubernetes chart
(cert-manager's default `Always` would rotate the CA key at renewal).
- Restore into another application name or namespace.
- The tenant's shared etcd DataStore runs with auth disabled, so any
cluster's datastore certificate can read its neighbours' prefixes. This
predates this change; the dump Job uses the same certificate the
cluster's kube-apiserver already holds.
### Screenshots
N/A — no UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Walked against the diff: no package under `packages/apps|extra` is added
or renamed, `packages/core/platform` and installer values are untouched,
the new CRD is in `strategy.backups.cozystack.io` (not typed by
terraform-provider), and `RestoreJob.spec.options` stays opaque JSON.
### Release note
```release-note
feat(backups): tenant Kubernetes applications can now be backed up and restored through the cozy-default BackupClass. A backup captures the cluster's etcd state and its control-plane PKI and PersistentVolume data, and warns when a control-plane CA would expire before the backup does. A restore rebuilds a deleted cluster with its original CA or rolls a live one back, and can restore the etcd state and the volumes separately via RestoreJob options `etcd` and `volumes`.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
- **New Features**
- Added Kubernetes application backup and restore support, including
etcd data and persistent volumes.
- The default backup class now routes Kubernetes applications to the new
strategy.
- Added an end-to-end walkthrough for creating a cluster, backing it up,
restoring it in place or after deletion, and cleaning up.
- Backup status can report expiring certificates, and restore validation
helps protect cluster identity and state.
- **Documentation**
- Added guidance on backup contents, restore behavior, and walkthrough
configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
2cf80c02a4 |
fix(clickhouse): keep failed backup attempts out of the bucket (#5229)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse BackupJob attempt that fails during upload leaves its local backup on the data volume, and with the upstream defaults the clickhouse-backup sidecar never prunes local backups and resumes every unfinished upload when it starts. On the next sidecar restart the failed attempt is uploaded to the bucket as an archive no Backup refers to, and while that resumed upload runs the sidecar answers new operations with 423, which burns the strategy Job's retries in seconds. The strategy itself only waited for the sidecar API to respond and then started its operation regardless. The sidecar now runs with `API_COMPLETE_RESUMABLE_AFTER_RESTART=false`, so a failed attempt is never published, and `BACKUPS_TO_KEEP_LOCAL=-1`, so at most the newest local backup is kept and it is removed once its upload succeeds. Backups already left on existing volumes are cleaned up by the first backup after the upgrade. The Altinity strategy now waits until the sidecar has no operation in progress before it starts a backup or a restore, and fails the attempt with a clear message instead of proceeding when the sidecar stays busy or unreachable. Waiting for an idle sidecar changes what a retried strategy Pod does. The Job retries a failed Pod up to twice, and a Pod can die after its operation was accepted (a dropped poll under errexit, an eviction, a node drain) while the sidecar keeps running it. Before, the retry hit 423 and failed; with the idle gate it would wait for the running operation to finish and start it again. For a restore that is destructive, because `restore_remote` drops every table it restores before recreating it, so the second run discards whatever was written after the first one finished. A retried restore therefore takes over a `restore_remote` of the same archive that is still running and waits for its result instead of starting another. Once an operation is accepted, polls tolerate up to six failures in a row before the attempt fails, and the sidecar losing the operation from its in-memory action log, or ending it cancelled, fails the Pod instead of leaving it polling. A retried backup works from the archive name the controller records for its BackupJob. That name is now `<application>-<YYYYMMDDHHMMSS>-<first 12 hex digits of the BackupJob UID>`, so two runs of one application started in the same second, as two Plans on one schedule are, never share an archive. It is exactly as long as the previous `clickhouse-<application>-<YYYYMMDDTHHMMSSZ>`, so an `s3PathOverride` prefix that fit before still keeps every object key within the 1024-byte limit. A retry takes over a `create_remote --resume` of its archive that is still running, counts an upload of its archive whose latest run ended in `success` as its own result instead of uploading the archive again, and otherwise resumes the upload. Restoring a Backup that records no archive still picks an archive by timestamp, and that path accepts only `clickhouse-<application>-<timestamp>` names, so it never picks a per-run archive. One limit remains. A finished `restore_remote` row is not taken over: the sidecar never trims its action log, so a `success` row of the same archive may be left by an earlier RestoreJob, and counting it as this one would report a restore that never ran. A retry that arrives after the restore it replaces had already finished, but before that Pod saw the result, therefore still runs the restore a second time. Upgrading changes the ClickHouseInstallation pod template, so ClickHouse pods with `backup.enabled` are rolled once. A backup Job started before the upgrade by a controller that named archives the previous way is not re-rendered and keeps writing that name, so the controller records the previous name when the new one is not on the Job. The local copy of the latest failed attempt stays on the volume until the next backup. Objects partially written to the bucket by an upload that fails midway are not removed by clickhouse-backup and are left for a follow-up. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): failed ClickHouse backup attempts are no longer uploaded to the bucket when the backup sidecar restarts, local backups no longer accumulate on the data volume, the backup strategy waits for the sidecar to be idle instead of failing with HTTP 423, a retried ClickHouse restore takes over the restore that is still running instead of starting it a second time, and a retried ClickHouse backup takes over or reuses its own upload instead of starting a new one. Every ClickHouse backup run now writes an archive of its own, even when two runs start in the same second, under a name no longer than before. ClickHouse pods with backups enabled are restarted once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup and restore operations wait for the backup service to become idle and join a matching operation already in progress rather than starting a duplicate. * Restore operations use the archive the Backup records and fail when it is not in remote storage; only a Backup that records no archive falls back to the newest eligible archive by timestamp. * New backups use distinct archive names, including for runs started in the same second. * Local backup copies are discarded after upload, and completed uploads are not resumed after a service restart. * **Bug Fixes** * Operations now fail when the service cannot be queried, an operation disappears from its activity log, or it ends with an error or cancellation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bc268ce759 |
fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first (#5152)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Disabling the Monitoring Agents addon of a tenant Kubernetes cluster leaves `<cluster>-monitoring-agents` (release `cozy-monitoring-agents`, installed into the tenant cluster) in `UninstallFailed` for good. Seen in production on 1.6.5 and still reproducible from `main`. `monitoring-agents`, `victoria-metrics-operator` and `vertical-pod-autoscaler-crds` in `packages/apps/kubernetes/templates/helmreleases/` are all gated on `addons.monitoringAgents.enabled`, so they disappear together and Flux uninstalls them in parallel: `dependsOn` is not honoured on deletion. Two things then break the uninstall of `monitoring-agents`: 1. The VictoriaMetrics operator is often gone first. The operator's `apps.victoriametrics.com/finalizer` stays on the `VMAgent`, nobody removes it, and the deletion hangs. The manual workaround was to strip the finalizer, after which GC removes the VMAgent's Deployment, Service, Secrets and ServiceAccount through their owner references. 2. The VPA CRD was deleted with its own release while the stored manifest still contains `VerticalPodAutoscaler vpa-vmagent`, so Helm cannot build the manifest to uninstall. That root cause is fixed by #3586 (`helm.sh/resource-policy: keep` on the tenant addon CRDs), which this PR does not redo. This PR: - adds a `pre-delete` hook (`templates/vm-cleanup-hook.yaml`) that deletes every VictoriaMetrics object the release owns, waits up to 60s for a live operator to finalize them properly, and clears their finalizers by hand when it has not. Objects are selected through Helm's `meta.helm.sh/release-name` / `release-namespace` annotations rather than by name, so the 34 `VMRule`s rendered from `alerts/` and the `VMServiceScrape` in `cozy-victoria-metrics-operator` are covered along with the VMAgent. The kinds handled are every VictoriaMetrics kind the chart renders: `VMAgent`, `VMRule`, `VMServiceScrape`, `VMPodScrape`, `VMNodeScrape`. Operator v0.68.4 only finalizes the VMAgent; the other kinds are covered in case another operator version finalizes them, and for them the patch is a no-op. A kind whose CRD is already gone is skipped, and the script always exits 0, since a failing pre-delete hook would itself block the uninstall. The Job is bounded by `activeDeadlineSeconds: 180` and `backoffLimit: 0`; its kubectl calls carry no `--request-timeout`, which would make kubectl skip the in-cluster credentials (`hack/check-hook-request-timeout.bats`) and reuses the digest-pinned `docker.io/clastix/kubectl:v1.32` image the etcd-operator hook uses, via a new `kubectlImage` value. The shape follows `packages/system/forklift/templates/forklift-cr-cleanup-hook.yaml`. - guards `templates/vpa.yaml` on `.Capabilities.APIVersions.Has "autoscaling.k8s.io/v1/VerticalPodAutoscaler"`, so a cluster that does not serve the CRD never records a VPA in the release manifest. Cheap defence in depth next to #3586. Points for review: - The hook's RBAC is a ClusterRole/ClusterRoleBinding, limited to `get, list, delete, patch` on the five `operator.victoriametrics.com` resources, rather than Roles. The objects span two namespaces, and a Role in `cozy-victoria-metrics-operator` could not be created if that namespace were already gone or terminating, which would fail the hook and the uninstall with it. All hook objects carry `hook-succeeded,hook-failed,before-hook-creation` except the Job, which keeps a failed pod for its logs (`ttlSecondsAfterFinished: 600`), as in the forklift hook. - When the operator is already gone, clearing the finalizer skips whatever the operator does on finalize, notably the cluster-scoped RBAC it creates for the VMAgent, which has no owner reference. That residue is the same the manual workaround left, and it is no worse than today's stuck release; removing it would mean hard-coding the operator's naming scheme for those objects, which I preferred not to do. - When the operator is alive, the hook only brings forward deletions Helm was about to make, so the operator's normal finalization still runs. Tests: `tests/uninstall_test.yaml` asserts that the hook is a `pre-delete` Job in the release namespace with a bounded runtime, that its RBAC is hook-scoped and grants exactly the verbs needed on every VictoriaMetrics kind the chart renders, that the binding targets the hook's ServiceAccount, that the script selects by release ownership, deletes, clears finalizers and exits 0, and that the VPA renders only when the capability is present (using `capabilities.apiVersions`). With the hook removed and `vpa.yaml` reverted, 6 of the 7 tests fail; with the fix, `helm unittest packages/system/monitoring-agents` passes 26/26. `hack/monitoring-agents-vm-cleanup-hook.bats` runs the rendered script against a fake `kubectl` that keeps the objects in a state file, with fake `date` and `sleep`. It checks that only objects annotated for this release are deleted and patched, that with the operator gone the hook polls through the grace period and then clears the finalizers, that a live operator's objects are never patched, that a missing CRD still exits 0 and cleans the other kinds, and that a release with nothing left exits 0 without deleting anything. Removing the ownership condition, making the patch a no-op, or inverting the polling condition each turns this file red. The hook has not yet been run on a live tenant cluster. Backport: labelled `kind/backport` (release-1.7). The release-1.6 backport is done by hand in #5158. The commit cherry-picks onto `release-1.6` with a single trivial conflict in `values.yaml` (context only: 1.6 lacks the `prometheus-node-exporter` resources block the new `kubectlImage` block sits after on `main`). Resolved locally, the chart's tests pass on 1.6 (9/9); 1.6's chart renders the same VictoriaMetrics kinds. Note that `release-1.6`'s monitoring-agents `Makefile` has no `test:` target, so CI does not run this chart's unit tests on that branch. ### Screenshots Not a UI change. ### Downstream repositories I walked the trigger map against the diff (a system chart's templates, `values.yaml` and tests): no package added or renamed, no platform values, no app schema, no CRD, no tooling or convention change. I also searched cozystack/website for a documented VMAgent finalizer workaround that this would make obsolete and found none. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(monitoring-agents): disabling the Monitoring Agents addon of a tenant Kubernetes cluster no longer leaves the monitoring-agents release stuck in UninstallFailed when the VictoriaMetrics operator is removed first; a pre-delete hook now releases the VictoriaMetrics objects' finalizers, and the VPA is only rendered when its CRD is served. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * VictoriaMetrics resources belonging to a release are cleaned up during uninstall, including resources that remain because of finalizers. Cleanup is limited to resources owned by that release. * Vertical Pod Autoscaler resources are rendered only when the cluster supports the required API. * **Configuration** * Added an image setting for the uninstall cleanup job, including a pinned image version and pull policy. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
dae76d186c |
Merge branch 'main' into fix/clickhouse-idle-before-backup
main pins the archive name a backup Pod keeps across a wait for late bucket credentials, against the controller's archive helper. That helper now names a run's archive with part of the BackupJob's UID, so the merged test has to give its BackupJob one. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
238b3ee491 |
fix(seaweedfs): enforce S3 bucket lifecycle rules (#5228)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does SeaweedFS accepts and stores an S3 bucket lifecycle configuration, but only the `s3_lifecycle` job of its plugin worker applies it, and that job is scheduled by the admin server. Cozystack deployed neither, so a lifecycle rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike. That also made the documented way to bound backup storage a no-op. This PR adds the admin server and a worker to Simple and MultiZone SeaweedFS instances, behind a new `lifecycle.enabled` value that is **off by default**. A Client instance is unchanged, because its rules are enforced where the instance it connects to runs. The worker registers for `s3_lifecycle` only, so vacuum, balancing and erasure coding are not started as a side effect. The admin keeps its job state on a 1Gi volume on `lifecycle.storageClass`, or on the cluster default when that is empty; `volume.storageClass` is not reused, because in MultiZone it can name a class local to one zone or node. The admin UI is protected by a generated password, because a tenant namespace admits traffic from the whole cluster. The dashboard Role lists the two new WorkloadMonitors. Enabling the worker alone was not enough on SeaweedFS 4.45. Its chart renders no `[grpc.s3]` section, so the s3 gateway served its internal gRPC in plaintext while the worker dials it with mTLS, and every lifecycle delete failed the TLS handshake. The section is backported from upstream (seaweedfs/seaweedfs#11157, released in 4.46) as a chart patch, applied with `--forward` so the next `make update` stops at the point where the patch must be dropped instead of silently reversing it. The s3 pods are restarted once so they load the section. This also repairs IAM cache propagation between s3 replicas, which failed in the same way, and it applies whether or not lifecycle is enabled. The data-plane release now retries a failed install or upgrade in place (`RetryOnFailure`) instead of rolling it back. With the previous default remediation and unlimited retries, an upgrade that waited out its timeout, for example on a tenant quota with no room for the admin volume, rolled back and retried forever, restarting the s3 pods twice per cycle. The worker's hostname anti-affinity is relaxed to a preference so a rollout finishes on a single-node cluster. #### Why lifecycle is off by default The daily walk that SeaweedFS uses to expire objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when no identity is given; it does not re-check the object's age either. A key overwritten between the walk's listing and the delete loses its new content. This path is unchanged from 4.45 through 4.52 and on upstream master (`weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go`, `identityMatches` in `weed/s3api/s3api_internal_lifecycle.go`). Backup buckets write unique keys and are barely exposed; a tenant bucket that rewrites a fixed key is. On top of that, turning enforcement on would start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so nothing changes for existing instances until `lifecycle.enabled` is set. Making it the default again is left to a separate PR that carries object identity on the walk path; upstream report: https://github.com/seaweedfs/seaweedfs/issues/12451. #### Turning lifecycle on and off When `lifecycle.enabled` is set, rules that have been stored but ignored start applying within minutes, and objects they have already expired are deleted on that first pass, so review existing bucket lifecycle rules first. Set `lifecycle.storageClass` before enabling, and prefer a replicated class: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. Moving the volume means disabling lifecycle, deleting the `admin-data-seaweedfs-admin-0` PVC, changing the class and enabling lifecycle again. Each instance gains two small pods and a 1Gi PVC; when the tenant quota has no room for them, the data-plane release keeps retrying in place until it has. When `lifecycle.enabled` is turned off again, the parent release removes the admin credentials Secret while the data-plane release removes the admin StatefulSet only on its own reconcile, shortly after. An admin pod that restarts in that window fails to start on the missing Secret until the StatefulSet is gone; nothing is lost, and the order is left as is. The admin PVC stays until the instance is deleted, and a later re-enable starts the admin with a new password. Verified on a test cluster with the 4.45 chart: without the backport, the worker found the expired objects and every delete failed with `tls: first record does not look like a TLS handshake`; with this branch and lifecycle enabled, a rule with a past expiration date removed the matching objects and left the others, both on a manual run and on the first scheduled pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up needed, not opened yet: `operations/services/backup-classes.md` (v1.7 and next) advises an S3 lifecycle policy, which the bundled SeaweedFS applies only once `lifecycle.enabled` is set; the SeaweedFS reference page gains `lifecycle.enabled` and `lifecycle.storageClass` from the regenerated README on the next release - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The terraform provider does not model the SeaweedFS application, so the new values do not reach it. ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): S3 bucket lifecycle rules can now be enforced. Simple and MultiZone SeaweedFS instances run the SeaweedFS admin server and a lifecycle worker when the new `lifecycle.enabled` value is set; it is off by default, so existing instances keep their current behaviour. It stays off by default because the upstream daily expiry walk deletes without checking that the object it listed is still the one stored, so an object overwritten in that window can be deleted although it is new. When enabled, rules that were stored but ignored start applying within minutes and objects they have already expired are deleted on that first pass; review existing rules first. Each enabled instance gains two small pods and a 1Gi volume, placed on `lifecycle.storageClass` or the cluster default. Independently of the switch, the s3 gateway now serves its internal gRPC over mTLS, which repairs IAM cache propagation between s3 replicas and restarts the s3 pods once on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added optional, daily S3 bucket lifecycle-rule processing for supported SeaweedFS configurations. It is disabled by default; Client instances rely on lifecycle rules configured on the connected SeaweedFS. * Added configurable storage for lifecycle job state and dashboard visibility for lifecycle components. * Enabled the S3 gRPC server to use its internal TLS identity when security is configured. * **Bug Fixes** * Improved storage-class handling so numeric-looking names remain strings and unset classes are omitted where appropriate. * Clarified backup cleanup guidance, including when lifecycle rules apply and how to reclaim storage. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3badbdef74 |
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
aa97451bc2 |
fix(platform): leave tenant-root's ingress empty when the root tenant has none (#5226)
## What this PR does Enabling SeaweedFS on the root tenant without ingress no longer hangs. `cozystack-basics` wrote `_namespace.ingress: tenant-root` for the root tenant whatever its `spec.ingress` said, so `seaweedfs-system` depended on an `ingress-nginx-system` release that nothing creates, and the platform backup bucket waited with it. Every other tenant already derives the key in `packages/apps/tenant/templates/namespace.yaml` and leaves it empty without ingress. `cozystack-basics` cannot read the root Tenant, so the platform now carries its ingress state the same way it carries monitoring: the new platform value `ingress.rootEnabled` reaches the `cozystack-values` Secret as `_cluster.ingress-enabled`, and `_namespace.ingress` is written only when it is true. The default is `true`, which keeps the value every existing cluster renders today, and a Secret written before the key existed keeps it as well. An operator running the root tenant without ingress sets `ingress.rootEnabled: false`, kept in sync with `Tenant/root spec.ingress` by hand, as with `monitoring.rootEnabled`. With the key empty, `seaweedfs-system` drops the dependency through the check it already had (`eq $ingress .Release.Namespace`), and nothing reads live cluster state for it. An empty `_namespace.ingress` now reaches tenant-root and every tenant inheriting from it, so the consumers that would break on it are adjusted in the same PR. The API rejects `ingressClassName: ""` (checked with a server-side dry run), and two consumers quote the value: - `kubernetes`: the proxied ingress-nginx addon Ingress is not rendered without an ingress; its Service still is. - `monitoring`: Grafana's `spec.ingress` is left out, since grafana-operator copies it into the Ingress verbatim. - `seaweedfs`: the S3 Ingress is disabled instead of rendering with no controller to serve it. The `KamajiControlPlane` `network.ingress` block is unchanged on purpose: its `hostname` is what the Kamaji provider uses as the control plane endpoint, and Kamaji itself skips an empty class name. Harbor, bucket, bootbox and alerta write the class unquoted, so an empty value renders as an unset class, which the API accepts. Tests: `packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml` pins the flag reaching `_cluster` both ways and with `ingress` set to null; `packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml` pins `_namespace.ingress` for a missing, string, bool and `"false"` flag, and that the other keys stay when ingress is off; `packages/extra/seaweedfs/tests/ingress_dependency_test.yaml` pins the dependency and the S3 Ingress for an own, empty and inherited ingress; the kubernetes and monitoring suites pin the gated Ingress. Reverting any of the gates, or flipping the missing-key default, fails the suites. Not covered here: - `_namespace.etcd` has the same hardcoded shape (#4783) and is left for its own change. - Toggling `Tenant/root spec.ingress` without changing `ingress.rootEnabled` leaves the value stale, as `monitoring.rootEnabled` does today. ### Screenshots Not a UI change. ### Downstream repositories The diff adds a key to `packages/core/platform/values.yaml`, which the website documents by hand. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#800 documents `ingress.rootEnabled` in the platform Package reference - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): a root tenant without ingress no longer leaves seaweedfs-system waiting for an ingress-nginx-system release that is never created. Set the new platform value ingress.rootEnabled to false to match Tenant/root spec.ingress: false. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added a root-tenant ingress setting, enabled by default, that can be turned off. * **Bug Fixes** * Kubernetes and Grafana ingress resources are now rendered only when ingress is configured. * SeaweedFS S3 ingress is enabled only when ingress is configured. * Ingress-dependent components now respect the root-tenant setting, while other platform services remain available when root ingress is off. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6a93765351 |
fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix
seaweedfs-cosi-driver v0.3.1 grants access by rewriting the legacy /etc/iam/identity.json. Each S3 gateway migrates that file to per-identity files and renames it away whenever it changes, so a grant written while a gateway is migrating is lost: the BucketAccess reports accessGranted and its Secret is written, but every gateway rejects the key with InvalidAccessKeyId and never recovers. Users added to a Bucket in one change lose a key regularly. v0.4.0 also writes the per-identity file on grant. The rest of v0.4.0 does not reach the platform. The readwrite-governance-bypass access policy is opt-in and no BucketAccessClass uses it. The refusal to delete a bucket that still holds locked objects applies only to object-lock buckets, and their classes are Retain, so the sidecar never calls DriverDeleteBucket for them. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
67fcadf689 |
fix(seaweedfs): disable the S3 Ingress in a tenant without ingress
The S3 Ingress kept rendering with no ingress-nginx to serve it. Now that seaweedfs-system no longer waits on ingress-nginx-system there, the release completes and the unserved Ingress would stay behind. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
275c840a8c |
fix(monitoring): leave Grafana's ingress out in a tenant without ingress
grafana-operator creates the Ingress from spec.ingress verbatim, so an empty _namespace.ingress reached the API as ingressClassName "" and was rejected. With no ingress-nginx in the tenant there is nothing for that Ingress to attach to. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c61e6a842e |
fix(kubernetes): skip the proxied addon Ingress in a tenant without ingress
An empty _namespace.ingress means no ingress-nginx serves the tenant, which tenant-root now reports when the root Tenant has ingress off. The Ingress would carry ingressClassName "", which the API rejects, and the release would fail to install where it used to leave an unserved Ingress behind. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
b26001b6f9 |
test(seaweedfs): pin the ingress-nginx dependency to the namespace's ingress
seaweedfs-system waits on ingress-nginx-system only when _namespace.ingress names its own namespace. With tenant-root's key now empty when the root Tenant runs no ingress, an empty value has to drop the dependency, and a value inherited from a parent must not add it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5b6708c7f9 |
fix(platform): leave tenant-root's ingress empty when the root has none
cozystack-basics wrote _namespace.ingress: tenant-root whatever the root Tenant's spec.ingress said, so applications in tenant-root were told an ingress-nginx runs there when nothing deploys it. SeaweedFS then waited forever on ingress-nginx-system, and the platform backup bucket with it. Every other tenant already derives the key and leaves it empty without ingress. cozystack-basics cannot read the root Tenant, so the platform carries its ingress state the way it carries monitoring: ingress.rootEnabled reaches the Secret as _cluster.ingress-enabled and gates the key. The default keeps the value existing clusters render, and a Secret written before the key existed keeps it too. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
9978771858 |
feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does Adds `observability-proxy`, a new system component that lets a tenant user read the logs of their own namespace from VictoriaLogs with the same Kubernetes credentials they already use for the console, and with Kubernetes RBAC deciding what they can read. This is the first part of the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94); this PR implements the namespace logs root only. Metrics and the per-application roots follow in separate PRs. Why a proxy: today the log stores are shared per root tenant and have no notion of a caller, so tenants cannot be given direct read access without seeing other tenants' logs. The proxy authenticates the caller, authorizes the request from its path, and rebuilds the upstream query so that its scope cannot be widened by the client. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. What is included: - `cmd/observability-proxy` and `internal/observabilityproxy`: the proxy. - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>` for the VictoriaLogs read endpoints the console needs (`query`, `hits`, `stats_query_range`, `tail`, `field_values`, `stream_field_values`, `facets`). Anything else under the root answers `404`. - Authentication by `TokenReview`, authorization by one `SubjectAccessReview` built from the path (`list` on `cozystack.io/logs` in the namespace), both cached briefly. - The upstream request is rebuilt from an allowlist of parameters; the namespace scope is always forced as `extra_stream_filters`. Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL `options(...)` block, and tenant headers such as `AccountID`/`ProjectID` are not forwarded or are refused. - Non-canonical paths (`..`, `//`, encoded separators) are refused with `400` before any authorization is built, because the console nginx matches on the normalized path but forwards the raw one. - Time window, step, bucket count and result limits are clamped; per-user rate, concurrency and open-tail limits answer `429`. Invalid time, step and limit parameters answer `400`. - The store is resolved from the namespace's `namespace.cozystack.io/monitoring` label. - Live `tail` streams are passed through unbuffered, and the upstream tail request disables compression so headers reach the client before the first line. - `packages/system/observability-proxy`: chart and image build, wired into the root `make build`. - Platform wiring: a new `cozystack.observability-proxy` PackageSource, enabled by default on the full variants and opt-in on the slim ones, the same way as `monitoring-agents`. - RBAC: an aggregated `cozy:observability:view` ClusterRole grants `list` on `cozystack.io/logs` to tenant `view` and above. No API server serves this resource; the rule exists only for the proxy's `SubjectAccessReview`. The proxy's own ClusterRole is limited to the calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch` on namespaces). - Console nginx: when the proxy is not installed or cannot be reached, the observability roots answer `404` with a Kubernetes `Status` body, which the console shows as not available on this cluster; the proxy's own `502`/`504` for an unreachable store pass through unchanged. - Console: one regex `location` in the console nginx routes the four observability roots to the proxy ahead of `/apis`. The upstream is resolved at request time, so the console starts even when the proxy is not installed; those paths then answer `404` with a Kubernetes `Status` body. The console Deployment now receives the cluster domain for that resolution. - E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/` with two tenants, covering `401`/`403`/`200`, scope-widening attempts, path normalization and a long-lived `tail`. #### How it was tested - `go test -race` and `go vet` for the proxy packages. - helm-unittest for `observability-proxy`, `dashboard`, `tenant` and `core/platform`; bats tests for the console nginx routing. - The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs as the root store and refuses to run when a real one exists. Its cases were replayed by a script against the real store on a test cluster. - Verified on a test cluster: - `401` without a token and with an invalid token, `403` for another tenant's namespace, `200` for the caller's own. - Client `extra_stream_filters`/`extra_filters` (GET and POST), an `AccountID` header and another tenant's stream named inside the query do not widen the scope; `options(...)` answers `400`; a path with `..` pointing at another tenant answers `400`; an endpoint outside the allowlist answers `404`. - Through the full browser chain (OIDC login, gatekeeper, gateway, console nginx), a single `tail` stayed open for more than two minutes without reconnecting, and requests from the same session to other tenants' roots answered `403`. - Not verified: the `429` limits under real load. `golangci-lint` has not been run locally. #### Known limitations - The proxy is reachable only through the console route; tenant workloads have no egress rule to it. Every request takes its concurrency or tail slot before the body is read; the token and access reviews and the body read run under `--request-timeout`, so a stalled kube-apiserver answers `503` rather than holding the request; and writes to the client carry a deadline (renewed per line for tails). `--request-timeout=0` disables those bounds and should not be used in production. - The first lines a newly started pod writes can be missing from an already open `tail`, although they are in the store. The likely cause is that Fluent Bit picks up a new log file later than the VictoriaLogs live-tail window; later lines of the same pod arrive. A history query shows them. - The per-application roots (`.../<plural>/<name>/log`, `.../metrics`) and the namespace metrics root answer `404` in this PR. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/787 (docs land once the implementation PRs merge) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: `cozystack/website` is reached. A new platform component (`packages/core/platform/sources/observability-proxy.yaml`) triggers `content/en/docs/next/guides/platform-stack/_index.md` and `operations/configuration/licenses.md`; adding it to the system bundle (`packages/core/platform/templates/bundles/system.yaml`) may also need a line in `operations/configuration/variants.md`. A user-facing page for the logs API and the new tenant `view` permission is not required by the trigger map but would be useful. Tracked in cozystack/website#787. Other repositories: no trigger matched. `packages/core/platform/values.yaml` is not changed (only a test fixture), no package under `packages/apps/` gains a values field, and no CRD changes. The root `Makefile` `build` target builds one more image; this does not touch `hack/` or the make targets `ccp` relies on, so `ccp` is not considered affected, but a reviewer may want to confirm. ### Release note ```release-note feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added an observability proxy for tenant-scoped log access through the console, including live log streaming. Requests are authenticated, access-controlled, and subject to configurable rate and concurrency limits. * Added platform packaging and deployment support, with console routing for observability requests. * **Tests** * Added automated coverage for access control, tenant scoping, query handling, streaming, proxy behavior, and deployment configuration. * **Documentation** * Updated end-to-end test guidance with host requirements for running the observability proxy suite. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8ba0380043 |
docs(clickhouse): name the clickhouse-backup handlers that reload the config
clickhouse-backup reloads its config file in each handler that reaches S3, among them the create_remote, restore_remote and list calls the strategy makes. The status and action-log endpoints the strategy polls read in-memory state and do not reload it, so the mounted file reaches the sidecar on the calls that need credentials rather than on every request. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f5114ae04c |
fix(seaweedfs): keep the metadata database storage class a string
db.storageClass went unquoted into the values of the database release and again into its CNPG Cluster. A class name made only of digits, which Kubernetes allows, became a number that the Cluster rejects, and a line break in the value could add keys to the database release values. It is quoted at both steps. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e6eb3345c6 |
fix(seaweedfs): escape pool and zone storage classes in the release values
Pool and zone storage classes were wrapped in quotes by hand, which does not escape a quote or a backslash in the value: a quote closes the string early and the data-plane release values stop parsing, with a YAML error that does not name the field. They are now quoted like the volume and admin classes. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f83db514b0 |
fix(seaweedfs): quote every storage class the upstream claims carry
The master, filer, volume index, log and all-in-one claims of the upstream chart write storageClassName unquoted, like the data claims did, so a class name made only of digits renders as a number and the claim is rejected. None of them is set by Cozystack today, but they are values of the same chart that a later change can start handing on, and the patch already exists for the data claims. They follow the same rule: quoted, and left out when no class is named so that the cluster default applies. The deprecated filer.enablePVC claim is left alone: it renders only with filer.data unset, and the template fails earlier on an unset filer.data, so it cannot be reached. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e7b5a5a1a9 |
fix(seaweedfs): keep storage class names strings in the volume claims
The upstream admin and volume StatefulSets write storageClassName unquoted, so a class name made only of digits, which Kubernetes allows and the parent chart now hands on as a string, turned back into a number there and the claim was rejected. The field is quoted, and left out when no class is named: quoting an empty value would set an empty storageClassName, which disables dynamic provisioning instead of using the cluster default. Upstream 4.52 and master still write both fields unquoted, so this is a chart patch, applied with --forward so make update stops once upstream carries an equivalent change. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7c9aa2fe10 |
fix(seaweedfs): hand storage class names to the data plane as strings
The volume and admin storage classes were written unquoted into the values of the data-plane release. A class name made only of digits, which Kubernetes allows, arrived there as a number, and a value with a line break could add keys next to the field, past the schema that is the tenant's input gate. Both are now quoted. lifecycle.storageClass, new in this release, also carries the StorageClass name rules in its schema, so a value that cannot name a class is refused when the application is written instead of leaving the admin claim pending. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ba0312b35f |
chore(kubernetes): align monitoring fix with current platform base
The previous base differs from the published package configuration, so the image overlay retains release controllers instead of current builds. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0f44cf9e25 |
feat(backups): add Kubernetes backup strategy for tenant clusters
A tenant Kubernetes application had no backup route in cozy-default, and none of the existing strategies fits it. Its API state is a key prefix in the etcd DataStore the whole tenant shares, so the Etcd strategy's snapshot would roll every tenant cluster back at once; its control-plane CA, service account key and Talos secrets cannot be regenerated without breaking every node and credential issued under them; and its PersistentVolumes are DataVolumes in the tenant namespace that the application's own deletion removes. The new Kubernetes strategy backs up the cluster's etcd prefix with a Job that authenticates with the cluster's own datastore certificate, and the PKI Secrets and data DataVolumes with Velero. The HelmReleases of the cluster and its node pools are recorded on the Backup rather than in Velero, because a label selector for them would also pick up the worker system disks. A backup warns when a CA would expire before the backup does. A restore rebuilds a deleted cluster under its old CA, or rolls a live one back after stopping its control plane and workers, and lets the etcd prefix and the volumes be restored separately. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-by: LLM |
||
|
|
19dbee15fc |
docs(observability-proxy): explain why both application roots need the namespace root
The reason given for the second access review held only for the metrics root. The log root needs it too, because an ownerless pod keeps the lineage labels its creator set and can claim any application. The tenant ClusterRole comment also named only the namespace roots, though the application roots now depend on the same rule. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
85fb98dd2c |
fix(observability-proxy): stop refusing ordinary app queries on lookback
With the lookback refusal, the 24h default turned common application
queries into 422: a day's increase, and month-long panels at the step
a dashboard picks for them. The lookback estimate is a loose upper
bound, made for the store choice, and counts every "(" as an implicit
subquery, so it far exceeds what such queries read.
The lookback is only the refusal threshold, since the lookup itself
spans the window the query reads, so its default now matches
--metrics-max-range. The chart passes it from values, so an operator
can tune it.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
|
||
|
|
2f600d5354 |
feat(observability-proxy): serve the application logs root
A viewer of one application gets that application's lines without writing a namespace query and filtering by hand, and a grant narrowed to one application by resourceNames is honoured on every logs endpoint, tail and the pickers included. The namespace filter stays and the application's identity is ANDed next to it as a second stream filter, so the result can only be narrower than the namespace root. The plural is mapped to its kind only after the caller is authorized, so an anonymous client cannot probe which applications exist, and the proxy is not ready until the ApplicationDefinitions cache is. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
5630de4ab1 |
feat(observability-proxy): grant tenant viewers the metrics root
The proxy authorizes the namespace metrics root with a SubjectAccessReview for list on cozystack.io metrics, which no role grants yet. Aggregating it into the tenant view role reaches every tenant group and ancestor subject without a tenant chart change. The Deployment passes the metrics store names and short-term retention from values, so a platform that renames or resizes its metricsStorages can keep the proxy's store choice in step. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
b53ca1fede |
fix(postgres): refuse system databases and never drop them (#5155)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
`databases` accepts `postgres`, `template0` and `template1`, and the
init job treats them like application databases: it comments them
`database managed by helm`, creates `<db>_admin` and `<db>_readonly`,
makes `<db>_admin` the owner of the database and its schemas, and
installs the `trigger_auto_grant` event trigger. When the entry later
leaves values, the delete pass runs `DROP DATABASE` on it, which can
only fail:
- `template1`, `template0`: `cannot drop a template database`;
- `postgres`: `cannot drop the currently open database`, since the job
is connected to it.
The init job then fails on every upgrade, so the HelmRelease sits in
`UpgradeFailed` with `post-upgrade hooks failed`, and putting the entry
back does not help. `template1` also hands the takeover (owner of
`public`, default privileges, trigger) to every database created
afterwards. Once its roles are dropped by hand, the trigger left behind
names a role that no longer exists, so `CREATE SCHEMA` fails in every
new database, even as superuser.
This PR:
- **Refuses** the three names at render with a message that says what to
do. `cozyvalues-gen` has no way to constrain the keys of a
`map[string]Database`, so the refusal sits next to the existing
database-name guard, like the `users.postgres` refusal. The field
description says it too.
- **Never selects them for `DROP DATABASE`.** The delete pass excludes
them whatever values say.
- **Hands a taken-over system database back.** In a new `release system
databases` pass before the delete pass, for each of the three still
commented `database managed by helm`: drop the trigger and
`auto_grant_schema_privileges()`, `REASSIGN OWNED` / `DROP OWNED` for
its two roles in that database, give `public` back to
`pg_database_owner` (PostgreSQL 15+), and clear the comment. The
existing orphaned-roles pass then drops the roles; it never visits
`template1`, hence the per-database step. `template0` accepts no
connection and the job that commented it never got further, so only its
comment is cleared.
**Upgrade note.** A release that declares one of the three names fails
to render until the entry is removed. Removing it is now safe: the next
run releases the database instead of dropping it.
**Testing**
- `tests/system_databases_test.yaml`: 5 of its 6 cases fail on `main`
without the change. `helm unittest packages/apps/postgres`: 215 passed.
- I replayed the scenario against a `postgres:17` container, running the
rendered `init.sh` with `PGUSER=postgres`:
1. The `main` script with `databases: {appdb, template1, postgres}`
reproduces the takeover.
2. The `main` script with only `appdb` then fails with `cannot drop the
currently open database`.
3. This branch's script on that same state exits 0. It releases both
databases (owner `postgres`, no comment, no event trigger, `public`
owned by `pg_database_owner`) and drops `postgres_admin`,
`postgres_readonly`, `template1_admin` and `template1_readonly`. A
database created afterwards gets no trigger, and `CREATE SCHEMA`
succeeds in it.
4. A second run is a no-op.
5. A `template0` commented and given roles by hand is released the same
way.
Open PR #5091 touches the same script (stable database order); the hunks
do not overlap.
Backport: the automatic cherry-pick onto `release-1.6` conflicts in
`init-script.yaml` and the generated cozyrd, so the backport to
`release-1.6` is done by hand in #5157 (same SQL, replayed against that
branch's script). `release-1.7` keeps `kind/backport`.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change (walked the
map: a template guard, the init script, a test suite and one field
description regenerated into README, schema, Go types and cozyrd; no
field, default, enum or name change)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(postgres): databases now refuses the PostgreSQL system databases postgres, template0 and template1, and the init job never runs DROP DATABASE on them. A system database a previous version took over is handed back to the superuser on the next upgrade (event trigger removed, ownership and public schema restored, its <db>_admin and <db>_readonly roles dropped). A release that declares one of them must remove the entry before upgrading.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Behavior Changes**
* PostgreSQL system databases (`postgres`, `template0`, and `template1`)
are refused as managed application databases.
* Previously managed system databases are released during cleanup and
excluded from database deletion. Cleanup restores eligible ownership and
security settings and clears system database comments.
* **Documentation**
* Updated PostgreSQL configuration guidance to clarify that these system
database names are refused. Names that merely contain these strings
remain accepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
f2944c572d |
fix(dashboard): answer not-installed when the observability proxy is absent
The proxy can be left out of an installation, and then the console nginx answered the observability roots with its own 502. The console read that as an unreachable store: three errors, a Retry button and a live tail reconnecting forever, instead of a notice that the feature is not available on this cluster. nginx now turns the 502 and 504 it generates itself, when it cannot resolve or reach the proxy, into a 404 Kubernetes Status, which the console already treats as not installed. Upstream errors are not intercepted, so the proxy's own 502 and 504 for a store still reach the console unchanged. The bats test runs the extracted locations in a real nginx when one is installed. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
2b6a1f4548 |
test(observability-proxy): drop the tenant egress policy from test notes
The chart tests named the tenant egress policy as one of the parties the pod label and port are pinned for. That policy is gone, so the notes now name only the Service and the console nginx, which still depend on them. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
b703a54cb4 |
fix(tenant): drop the tenant egress rule to the observability proxy
The rule was added for a tenant Grafana reading through the proxy, but the tenant Grafana datasources still point straight at vlselect and vmselect, and the console reaches the proxy from cozy-dashboard. The rule therefore served no caller and only let every pod of every tenant reach the proxy directly, outside the console route, where each request with a fresh token costs a TokenReview. It can come back together with a tenant client that needs it. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
889712d244 |
fix(observability-proxy): grant the proxy only the API calls it makes
Nothing the proxy does reads ApplicationDefinitions, so the rule for them is dropped. Namespaces are read through an informer, which only lists and watches, so get is dropped as well. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
91f03dd90e |
fix(platform): remove the dead registries value and patch-containerd Secret (#5190)
Nothing reads the `patch-containerd` Secret: the kubeadm certs.d mount it fed is gone, and the v1.6.0 changelog already says the passthrough is removed. The `registries` value stayed in the Package spec and takes mirrors that do nothing, which is worse than no knob for someone setting up an air-gapped cluster. Removed both. A Package that still sets `registries` keeps installing, the platform chart has no values schema so the key is ignored. On upgrade the Secret in `cozy-system` is deleted by the release. Mirrors for tenant workers are the per-pool `talos.registryMirrors` of KubernetesNodes. I checked that nothing is left: `git grep` for `.Values.registries` and for `patch-containerd` outside the changelogs finds nothing but the kubernetes README sentence that says the Secret was removed. Full `make unit-tests` passes. Fixes https://github.com/cozystack/cozystack/issues/4006 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration Changes** * The platform chart no longer supports the `registries` configuration option or generates containerd registry secrets. Mirror endpoint settings, including credentials and TLS verification options, are no longer applied through this chart. * Registry mirrors can be configured per worker pool using `talos.registryMirrors` on `KubernetesNodes`. Existing packages that set `registries` continue to install, but the setting is ignored. * **Documentation** * Updated guidance for configuring registry mirrors on worker pools. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d530ca8cf5 |
fix(backups): keep a ClickHouse archive an earlier attempt finished
A backup Pod that loses sight of its upload, by dying or by losing six polls in a row, is retried while the sidecar finishes that upload. The retry found it in the action log as a success, took the sidecar for idle and asked for create_remote with resume again. clickhouse-backup then rebuilds the local backup and uploads it once more over the archive already in the bucket, which is left broken if that second upload dies halfway. The archive name belongs to one BackupJob, so a success row of exactly this upload can only be an earlier attempt's, and the retry now takes it as its result. An error or cancel row still leads to a resumed upload, and a restore still runs again, since a restore is not named per RestoreJob. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
89f8cc02ee |
Merge branch 'main' into fix/clickhouse-idle-before-backup
main now names the ClickHouse archive in the controller and restores the archive a Backup recorded, in the same strategy script this branch gates on an idle sidecar. The merged script keeps both: the archive name and its restore selection from main, the idle gate, retried polls and lost-operation check from this branch. A retried backup is taken over through the same idle gate as a restore, matched on the whole command line the sidecar logs. With resume requested that line reads "create_remote --resume <name>", so the gate matches that form; matching on the name alone would also take another operation's row ending in the same name for this one. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
4f35af6694 |
fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first
Disabling the Monitoring Agents addon of a tenant cluster removes this release, the VictoriaMetrics operator and the VPA CRDs together, and Flux deletes them in parallel without honouring dependsOn. When the operator goes first, nothing clears apps.victoriametrics.com/finalizer on the VMAgent and the other VictoriaMetrics objects this chart created, so they stay Terminating and the release sits in UninstallFailed for good. Clearing those finalizers has to happen inside the uninstall and must never be what blocks it, and a live operator still has to get the first chance to finalize its objects. Only objects of this release may be touched, and release names are unique only within a namespace, so the release name alone does not identify them. kubectl loads in-cluster credentials only while the client config is otherwise default, which a --request-timeout breaks. A VPA recorded in the release manifest while its CRD is not served cannot be mapped at uninstall either. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
f003851338 |
fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled like an application database. The init job made <db>_admin their owner, installed the auto-grant event trigger in them and, once the entry left values, ran DROP DATABASE on them. That DROP can only fail (template databases cannot be dropped, and the job is connected to postgres), so every later upgrade failed in the init job and the release stayed in UpgradeFailed even after the entry was put back. template1 also passed the takeover on to every database created from it afterwards, where the trigger, naming a role about to be dropped, makes CREATE SCHEMA fail. A release a previous version let take one of them over has to be repaired without dropping it, and without touching what this chart did not create: a role or owner that merely carries an expected name may belong to a tenant or an administrator, and a role comment can be set by any CREATEROLE holder, so only the cozystack:managed mark is trusted. What the marked roles own goes to cozystack:orphaned rather than to the superuser, since a tenant's SECURITY DEFINER function would otherwise run with superuser rights. A failed lookup of the databases cloned from template1 has to stop the run before template1 stops being selected, or those clones would never be repaired. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
da7f2c2f50 |
fix(tests): keep two unit tests green after the release image restamp (#5180)
## What this PR does Same change as #5179, on `main`. Both tests pass here today, because `main` still carries the pre-release image pins; they fail on every release line as soon as the first rc is cut, since the `Prepare release` commit rewrites `packages/*/values.yaml` and two unit tests read exactly the fields it rewrites. On `release-1.7` this has kept `Unit & controller tests` red on every PR since v1.7.0-rc.3 ([job](https://github.com/cozystack/cozystack/actions/runs/41162009892/job/124421449910) on #5172), and `release-1.8` would repeat it. | test | reads | before the release build | after (release-1.7 today) | |---|---|---|---| | `hack/cni-plugins-staging-contract.bats` "the kube-ovn release the CNI plugin pin was checked against is the one vendored" | `.global.images.kubeovn.tag` in `packages/system/kubeovn/values.yaml` | `v1.15.26@sha256:…` | `v1.7.0@sha256:…` | | `packages/system/monitoring/tests/vtraces_collector_test.yaml` "pins the collector image through the cozy-lib.image helper" | `tracingCollector.image` | `docker.io/otel/opentelemetry-collector-contrib:0.136.0@sha256:…` | `ghcr.io/cozystack/cozystack/otelcol-tracing:v0.136.0-cozystack.0@sha256:…` | Changes: - The bats test takes the vendored kube-ovn release from `appVersion` in `packages/system/kubeovn/charts/kube-ovn/Chart.yaml`. That is the value `packages/system/kubeovn/Makefile` builds the image from, and a release build leaves it alone. The hint printed on a mismatch names the new file. - The helm unittest case no longer names the image at all. It sets `_cluster.images-registry` and asserts that the registry lands in front of the rendered image and that the image stays digest-pinned (the `grafana_upstream_image_plugins_test.yaml` pattern). Removing the `cozy-lib.image` call from the template makes it fail; the old regex did not. Same single commit as #5179; the only difference is the unrelated `DOC=` line next to the edit, kept as `main` has it. Verified locally on this branch: `bats hack/cni-plugins-staging-contract.bats` 10/10, `helm unittest packages/system/monitoring -f tests/vtraces_collector_test.yaml` 28/28. Refs #5178 (the release-1.7 report), #5179, #5172. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Tests** * Version validation now compares the checked version with the chart’s application version and reports missing or mismatched versions with the chart path. * Monitoring image checks now verify use of the configured mirror registry and require a 64-character lowercase SHA-256 digest, without requiring a specific image name. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
23461e065d |
fix(redis): run one Sentinel when Redis runs a single pod (#5151)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The chart pins `spec.sentinel.replicas: 3` whatever `replicas` says, so an instance created with `replicas: 1` runs one Redis pod and three Sentinel pods that have nothing to fail over to. This makes the Sentinel count follow the data pods: one Sentinel when `replicas` is 1, three otherwise. The operator cannot run zero Sentinels (freshworks `Validate()` resets `sentinel.replicas <= 0` to 3), and it needs at least one to locate the master. With one Sentinel its quorum is `replicas/2+1` = 1, so the single Sentinel works on its own. The operator also gives the Sentinel Deployment a PodDisruptionBudget with `minAvailable: 1` whenever Redis has two replicas or fewer (`ensurePodDisruptionBudget` in `operator/redisfailover/service/client.go`). On a single Sentinel that budget refuses every eviction and would block node drains, so the chart sets `sentinel.disablePodDisruptionBudget: true` in that case only. The Sentinel `WorkloadMonitor` now expects the rendered count instead of a fixed 3/2. The operator only stops ensuring that PDB once the flag is set; it never deletes the one it created while three Sentinels ran (`EnsureSentinelDeployment` in freshworks v3.3.5). An existing instance scaled to one replica would keep it, so `templates/hooks/sentinel-pdb.yaml` adds a post-upgrade hook that deletes `rfs-<release>`, with a Role scoped to that one name. It renders on upgrade only, since a fresh one-replica install never gets that PDB. From `replicas: 2` up nothing changes. Scaling an existing instance between 1 and 2 replicas scales the Sentinel Deployment between 1 and 3 accordingly. Not in scope: with `replicas: 1` the Redis data pod keeps its own operator PDB (`minAvailable: 1`), which already blocks drains of its node today; that is unchanged here. Tests: `tests/sentinel_replicas_test.yaml` covers 1, 2, 3 and the default, the PDB switch, and the `WorkloadMonitor`. Three of its cases fail on `main` without the template change. `tests/sentinel_pdb_hook_test.yaml` covers the hook. `helm unittest packages/apps/redis`: 79 passed. Open PR #4990 also edits `redisfailover.yaml` (Sentinel spreading); the two touch different keys. Backport: `release-1.7` ships the same freshworks operator and CRD, so `kind/backport` applies. `release-1.6` is not backportable as is: it still runs spotahome `v1.3.0-rc1`, whose vendored CRD (chart 3.3.0) does not declare `sentinel.disablePodDisruptionBudget`, and the operator's CRDs are installed with Flux's default `crds: Create` policy, so they are never updated on an existing cluster. The API server would prune the field and the single Sentinel would keep a `minAvailable: 1` PDB that blocks node drains. A 1.6 fix would need a CRD update path first. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change (walked the map: the diff changes a template, a test, and the wording of two descriptions; no field, default, enum or name changes) - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(redis): an instance with `replicas: 1` now runs one Sentinel instead of three, without a Sentinel PodDisruptionBudget; on upgrade, the PDB the operator created earlier is deleted. Instances with two replicas or more are unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Redis deployments configured with one replica now use one Sentinel; other replica settings continue to use three. * Workload monitoring now reflects the configured Sentinel count. When changing to one replica, upgrades remove the Sentinel PodDisruptionBudget. * **Documentation** * Updated Redis configuration guidance to explain how Sentinel count varies with the replica setting and clarify Sentinel resource defaults. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d61db784a6 |
fix(mariadb-operator): derive the cluster name from the platform domain (#5123)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Fixes #2348. `packages/system/mariadb-operator/values.yaml` pins `mariadb-operator.clusterName: cozy.local`, and the vendored chart turns it into the operator's `CLUSTER_NAME` and into a SAN of the webhook certificate. On a platform installed with any other `networking.clusterDomain`, the operator resolves its own pods as `<name>.<ns>.svc.cozy.local` and gets NXDOMAIN. Observed on a cluster with `clusterDomain: lab.local`: a 2-replica MariaDB never gets replication configured (`status.replication` stays `{}`, the secondary runs as a standalone semi-sync master), and User, Database and Grant fail with `lookup mariadb-<name>-primary.<ns>.svc.cozy.local ... no such host`, while the operator only retries at verbosity 1. `clusterName` is a value of the vendored subchart, which the `_cluster` lookup used by package templates cannot reach. The platform now carries `networking.clusterDomain` under the `mariadb-operator` subchart key in the shared `cozystack-values` Secret, the same route `keda.clusterDomain` and `opensearch-operator.manager.dnsBase` already take. The literal in the package stays as the fallback for a render without that Secret, so a cluster on `cozy.local` renders exactly as before. Tests cover both ends of the path: `packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml` pins that the Secret carries the domain under the anchored subchart key and tracks a non-default value (it fails with the `apps.yaml` change reverted), and `packages/system/mariadb-operator/tests/cluster_name_test.yaml` pins that a domain set on that key reaches the operator's `CLUSTER_NAME` and the webhook certificate, with `cozy.local` kept as the standalone default. `make helm-unit-tests` passes locally. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(mariadb-operator): the MariaDB operator now uses the platform's networking.clusterDomain instead of a hard-coded cozy.local, so replication and User/Database/Grant reconciliation work on clusters with a non-default cluster domain. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * MariaDB now uses the configured networking domain as its cluster name and webhook certificate address. If no custom domain is configured, it continues to use the default cluster name and address. * **Tests** * Added coverage for default and custom domain settings, including their effect on the MariaDB cluster name and webhook certificate address. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
0c486263f9 |
fix(postgres): render the declared databases in a stable order (#5091)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The postgres init script lists the declared databases with `keys .Values.databases`, which returns them in Go's randomized map order. A release with two or more databases therefore renders a different `DEFINED_DBS` line on each upgrade, so the `<release>-init-script` Secret and the `checksum/config` annotation of the init Job change on upgrades that did not touch the databases. Sorting the keys makes the render a function of the values alone. Found while measuring a platform-wide values change on a production cluster (#5092): of 36 releases that ran Jobs again, 31 rendered a byte-identical manifest across the two Helm revisions and the other 5 were Postgres releases whose only difference was this ordering, e.g. `DEFINED_DBS="db_a db_b"` → `DEFINED_DBS="db_b db_a"`. This does not stop the init Job from running on every upgrade (it is a `post-upgrade` hook); the issue covers that part. Verification: the new `tests/init_script_deterministic_test.yaml` renders eight databases declared out of order and expects them sorted. Without the fix it failed 5 runs out of 5 with a different order each time; with it, `helm unittest packages/apps/postgres` passes (27 suites, 210 tests). ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(postgres): the init script renders the declared databases in a stable order, so an upgrade that leaves the databases unchanged no longer rewrites the init-script Secret. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Database initialization now lists configured databases in alphabetical order, providing consistent output across deployments. * **Tests** * Added coverage to verify the alphabetical ordering. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9e8e10c307 |
fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does Refs #5205. The S3 endpoint the platform backups use is copied twice, and neither copy is refreshed after `publishing.host` changes. The COSI sidecar writes `BucketInfo` once per `BucketAccess`, so the credentials Secret `bucket-cozy-backups-system-credentials` rendered from it keeps the old endpoint. The Velero BSL and the default Strategy CRs keep whatever endpoint the last render of `backupstrategy-controller` saw. The default objects gate only acted on missing objects, so backups kept going to the old host and nothing reported it. The gate now also forces a render when a copy is stale: - the bucket's `<bucket>-system` release, when the credentials Secret's endpoint differs from the COSI Secret it is rendered from; - `backupstrategy-controller`, when the endpoint an object was rendered with differs from the credentials Secret's. The chart stamps the rendered endpoint on the BSL and on every default Strategy as `backups.cozystack.io/endpoint`, because the endpoint itself sits at a different path in every kind. Objects without the annotation count as stale until a render stamps it; `MinForceInterval` bounds the forces. Stale credentials are forced before the BackupClass and the routed objects are read, so an absent BackupClass or an unreadable Strategy still fails the check but does not keep the old endpoint in the credentials Secret. A `BucketInfo` without an endpoint is not compared, since rendering from it would write an empty endpoint or fail the bucket release. Repairing missing objects keeps working while the credentials comparison is stale or fails, and a `systemSecretName` that is not derived from a COSI Secret (no `-credentials` suffix, or the raw COSI Secret itself) is not compared at all. RBAC is unchanged: the controller already holds cluster-wide `get` on Secrets. No new metric: nothing would consume it. What this does not do: the COSI Secret still holds the endpoint the `BucketAccess` was granted with, so after a host change the endpoint follows only once an operator deletes the `BucketAccess` and forces `bucket-cozy-backups` to recreate it (this issues new S3 keys). From there the gate replaces the remaining manual steps of the recovery described in the issue. Automatic re-grant is left for a separate issue, and the operator runbook goes to the website, hence `Refs` rather than `Fixes`. Known limitation: the credentials comparison runs only when `backupStorage.systemSecretName` is `bucket-<bucketName>-system-credentials`, the Secret the forced `bucket-<bucketName>-system` release renders. If an operator sets `bucketName` and `systemSecretName` inconsistently, the comparison is switched off rather than forcing a release that never rewrites the compared Secret, so a stale endpoint in that Secret is not repaired. Repairing a missing credentials Secret still forces the release derived from `bucketName`, as before this PR. Tested with unit tests for the gate (including the comparison normalisation shared with the chart helpers), helm-unittest for the annotation and the wiring, and a bats guard that every rendered Strategy CR and the BSL carry the annotation. On a live cluster, a server-side render of the chart produced exactly the annotation the gate derives from the real credentials Secret, so the steady state does not force anything. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: an operator runbook for changing `publishing.host` with platform backups enabled (re-grant the `BucketAccess`, then let the gate converge; restart ClickHouse pods that read the credentials from environment variables). Not filed yet, pending approval of the text. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup strategies and storage locations now record their storage endpoint, enabling detection and repair of outdated endpoint settings after bucket access changes. * Bucket credentials and backup objects are refreshed independently when their endpoints are missing or out of date. * **Documentation** * Clarified when endpoint changes trigger updates and how suspended releases are handled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
74a7a44423 |
feat(dashboard): route the observability proxy roots from the console
The logs and metrics roots under /apis/cozystack.io and /apis/apps.cozystack.io are served by the observability proxy, not by the API server, so the console nginx sends them there ahead of its /apis prefix location, unbuffered and with the same long timeouts, so that a live tail reaches the browser as it is written. nginx resolves a literal upstream once at startup and refuses to start when the name does not resolve, which would take the console down on every cluster that does not run the proxy. The upstream is therefore a variable resolved per request, and such a cluster answers 502 on those paths only. That resolver does not apply the pod's search domains, so the config becomes a template the image's entrypoint renders with the pod's nameservers and the platform cluster domain, which the chart now passes to the console. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
f7719cf6e2 |
feat(tenant): allow egress to the observability proxy
A tenant's Grafana reads the tenant's logs and metrics through the observability proxy, and tenant egress is default-deny with one allow-to-* policy per platform service. The rule is narrowed to the proxy pods and their port, since the proxy's namespace holds nothing else a tenant should reach. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
bbdf05c6c7 |
feat(platform): deploy the observability-proxy
The proxy only answers for stores that the monitoring stack runs, so it follows monitoring-agents: on by default in the full variants and opt-in through enabledPackages on the slim ones, where monitoring is opt-in too. It is a stateless reader with no CRDs, so networking is its only dependency. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
7ec648d863 |
feat(observability-proxy): add the chart and image build
Tenants read their logs, and later their metrics, through an authorizing proxy that sits in front of the VictoriaLogs and VictoriaMetrics read paths, so the proxy needs a deployable package before the console or tenant Grafana can reach it. The proxy reviews every token and access decision against the API server, so its own role is limited to creating TokenReviews and SubjectAccessReviews and reading ApplicationDefinitions and namespaces. The namespace logs root is authorized as list on cozystack.io logs, which no API server serves; the rule is aggregated into the tenant view role so that a namespace viewer passes that review, as upstream grants pods/log at view. The image ref lands without a digest until the first build stamps it, and the package joins the root build list so every build path refreshes it. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
9fba8af69c |
test(redis): pin the release name in the Sentinel PDB hook for the longest release
The long-name case checked only that the Job name fits in 63 characters and that the Job targets the right PDB. A Job name that dropped the release, letting two releases' hooks replace each other, or a Role granting a different PDB would both have passed. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
e92bc8664b |
fix(tests): keep two unit tests green after the release image restamp
The release build rewrites packages/*/values.yaml: the kube-ovn image tag becomes the Cozystack version and the traces collector becomes the first-party otelcol-tracing build. Two unit tests read exactly those fields, so every PR against release-1.7 has failed Unit & controller tests since the first rc, and release-1.8 would repeat it. Same change as #5179 on release-1.7. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
a7f84242d1 |
fix(redis): keep three Sentinels for a zero or negative replica count
The operator runs three Redis pods when the count is zero or negative, but the chart treated any count up to 1 as the single-pod case. Such a cluster got one Sentinel and no Sentinel PDB, while three Redis pods ran under it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |