Aleksei Sviridkin
08c535c295
## What this PR does kube-apiserver's `/metrics` grows with the number of resources and CRDs, and on a full install it outgrows the 32MB `promscrape.maxScrapeSize` that monitoring-agents sets globally. vmagent then refuses every scrape of the API server, `up` drops to 0, and `KubeAPIDown` and `TargetDown` fire against a healthy control plane while every rule over API server metrics goes blind. The apiserver `VMServiceScrape` endpoint now sets its own `max_scrape_size` (128MB by default, `vmagent.apiserverMaxScrapeSize`). vmagent uses a per-job value in place of the global flag for that job only, so the 32MB guard for every other target is unchanged. The value stays finite on purpose: vmagent reads the whole uncompressed response into memory before parsing, `streamParse` included, so this limit is also a memory ceiling. A new `VMAgentScrapeSizeExceeded` warning fires on `vm_promscrape_max_scrape_size_exceeded_errors_total`, so an oversized target is reported as a size problem instead of only as a target that is down. The counter has no target label, so the description points at the vmagent targets page to find it. Upgrade note: the per-job value replaces the global flag in both directions. An install that raised `promscrape.maxScrapeSize` above 128MB through `vmagent.extraArgs` should raise `vmagent.apiserverMaxScrapeSize` as well. Verified on a dev cluster with the same operator (v0.68.4) and vmagent (v1.136.0): a test vmagent with a 1MB global limit scraped all three API servers with `max_scrape_size: 64MB` on the endpoint and refused all three without it. The new counter reached storage with the `namespace` and `pod` labels the alert uses, and the alert expression returned a series. The `VMRule` passes the operator's admission webhook, which rejects a malformed expression. Guest clusters get the same default, since their HelmRelease overrides other `vmagent` keys only. Fixes #4343 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(monitoring): kube-apiserver is scraped with its own 128MB size limit (`vmagent.apiserverMaxScrapeSize` in the monitoring-agents component) instead of the global 32MB, so a large `/metrics` no longer makes KubeAPIDown fire on a healthy API server. A new VMAgentScrapeSizeExceeded alert reports any target whose response exceeds the scrape size limit. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Kubernetes API server metrics can now use a configurable scrape-size limit, set to 128 MB by default. Leaving the setting empty uses the global limit. * **Bug Fixes** * Added a warning alert when vmagent reports scrape-size-limit errors. The alert identifies the affected vmagent pod and service and provides guidance for finding the target and adjusting its limit. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Cozystack
Cozystack is a free and open-source platform and framework for building clouds.
Cozystack is a CNCF Sandbox project that was originally built and sponsored by Ænix.
With Cozystack, you can transform a bunch of servers into an intelligent system with a simple REST API for spawning Kubernetes clusters, Database-as-a-Service, virtual machines, load balancers, HTTP caching services, and other services with ease.
Use Cozystack to build your own cloud or provide a cost-effective development environment.
Use-Cases
-
Using Cozystack to build a public cloud
You can use Cozystack as a backend for a public cloud -
Using Cozystack to build a private cloud
You can use Cozystack as a platform to build a private cloud powered by Infrastructure-as-Code approach -
Using Cozystack as a Kubernetes distribution
You can use Cozystack as a Kubernetes distribution for Bare Metal
Documentation
The documentation is located on the cozystack.io website.
Read the Getting Started section for a quick start.
If you encounter any difficulties, start with the troubleshooting guide and work your way through the process that we've outlined.
Versioning
Versioning adheres to the Semantic Versioning principles.
A full list of the available releases is available in the GitHub repository's Release section.
Contributions
Contributions are highly appreciated and very welcomed!
In case of bugs, please check if the issue has already been opened by checking the GitHub Issues section. If it isn't, you can open a new one. A detailed report will help us replicate it, assess it, and work on a fix.
You are welcome to take on the fix yourself. Commits are used to generate the changelog, and their author will be referenced in it.
If you have a concrete feature request, open an issue. For a cross-cutting or architectural change, open a design proposal in the community repo. For questions or early ideas, use GitHub Discussions. See CONTRIBUTING.md for the full routing.
Community
You are welcome to join us in #cozystack, on either CNCF Slack (get an invite) or Kubernetes Slack (get an invite), in our Telegram group, and at our community meetings. Add them to your Google Calendar or iCal for convenience.
Project documents
| Document | What it covers |
|---|---|
| Governance | How decisions are made and who makes them |
| Contributor Ladder | How to grow from contributor to maintainer |
| Maintainers | Who maintains the project |
| Code of Conduct | What we expect of each other |
| Security Policy | Reporting a vulnerability, and which releases get fixes |
| Adopters | Who runs Cozystack in production |
| Roadmap | Where the project is going |
License
Cozystack is licensed under Apache 2.0.
The code is provided as-is with no warranties.
Commercial Support
A list of companies providing commercial support for this project can be found on official site.