Timofei Larkin
436743eb34
## What this PR does The cozystack-controller meters S3 bucket sizes for billing by querying SeaweedFS `SeaweedFS_s3_bucket_(size|physical_size)_bytes` metrics from a VictoriaMetrics vmselect it locates via the `namespace.cozystack.io/monitoring` label on the tenant namespace. That discovery hardcodes the assumption that the monitoring stack scraping SeaweedFS runs in the same cluster. Some deployments host SeaweedFS — API and metrics both — in a separate cluster, so bucket sizes must be fetched from a remote Prometheus-compatible endpoint that discovery can never produce. ### Configuration surface A single controller flag, `--seaweedfs-metrics-endpoint`, exposed as a chart value: ```yaml cozystackController: seaweedfsMetricsEndpoint: "https://vm.example.com/path/to/prometheus" ``` The value is an arbitrary base URL of a Prometheus-compatible query API — scheme, host, optional port, and an optional path prefix — and the controller appends `/api/v1/query` to it, normalizing slashes so a trailing slash or a path prefix never produces a doubled or dropped slash. Basic auth can be embedded as URL userinfo (`https://user:pass@vm.example.com/...`); net/http turns it into an `Authorization` header on every request. TLS uses the system trust roots; a custom CA or bearer-token support would be a follow-up flag if a deployment needs it. The flag is validated and normalized at startup (absolute URL, http/https, no query string or fragment), so a malformed value fails the rollout instead of failing queries at runtime. Cluster-wide scope is deliberate: where SeaweedFS and its metrics live is a property of the deployment, not of any tenant, and tenant monitoring stacks keep working and keep being discovered via the `namespace.cozystack.io/monitoring` label for everything else — the flag only redirects the bucket-size queries. Being a deployment argument, it is settable exclusively by platform administrators, so no new writable surface is exposed to tenants. Rejected alternatives: a WorkloadMonitor spec field (bucket monitors are rendered by the tenant-facing bucket chart, so the field would have to travel through tenant-writable values — an SSRF primitive — and it would be near-permanent CRD surface for a deployment-level fact) and per-namespace annotations (per-tenant granularity that nothing needs, plus a second admin-facing configuration mechanism to document and validate). ### Backwards compatibility Strictly opt-in: with the flag unset — the default — resolution walks the exact same label-discovery path and produces the same vmselect URL. No existing deployment changes behaviour. ### Billing safety: failures no longer collapse to zero This also fixes a latent billing bug in the existing path: any metrics query failure (endpoint down, non-200, malformed response) produced an empty result map, and the reconciler overwrote `Workload.status.resources` with it — silently wiping the recorded bucket sizes, which billing reads as an empty bucket. Now: - `queryAllBucketMetrics` returns errors instead of swallowing them into an empty map, so "the bucket is empty" and "the endpoint could not be queried" are distinguishable. - On failure — and per bucket, when a query succeeds but reports no series for a bucket that previously had a size — the reconciler retains the last known `s3-storage-bytes` and `s3-physical-storage-bytes` values instead of dropping them. - The failure is surfaced three ways: a `BucketMetricsUnavailable` warning event on the WorkloadMonitor, an error-level log line, and an error returned from `Reconcile`, which lands in controller-runtime's reconcile error metrics for alerting. Unit tests cover URL validation and path joining, flag-over-discovery precedence, userinfo-based basic auth, and both retention scenarios (query failure and missing series). ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (controller Go code plus the cozystack-controller chart value and arg plumbing): none of the listed triggers match — no package added or renamed, no CRD, `ApplicationDefinition`, or platform values change, no `hack/` or layout change. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(workloadmonitor): a new --seaweedfs-metrics-endpoint controller flag (chart value cozystackController.seaweedfsMetricsEndpoint) points bucket size metering at an explicit Prometheus-compatible endpoint for deployments where SeaweedFS and its metrics live in a separate cluster; metrics query failures now retain the last known bucket sizes and surface an event and error instead of silently zeroing billing data ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an optional SeaweedFS metrics endpoint configuration. * Metrics endpoint URLs are validated and normalized before use. * Added support for endpoint overrides when discovering bucket metrics. * **Bug Fixes** * Preserved previously recorded bucket sizes when metrics are unavailable or incomplete. * Metrics failures now trigger retries and visible warning events without blocking other reconciliation. * Improved handling of invalid or unsuccessful metrics responses. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free platform and framework for building clouds.
Cozystack is a CNCF Sandbox Level Project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You can express your intention to on the fix on your own. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join our Telegram group and come to our weekly community meetings. Add them to your Google Calendar or iCal for convenience.
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.