mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-28 23:57:08 +09:00
feat(monitoring): bound every traces store by disk usage
The distributed tracing design requires each traces backend to be disk-bounded, not only age-bounded, so a full volume cannot block ingest. retentionDiskUsageBytes was optional, so a store without it filled its volume and stopped taking spans; with a shared-central store that stops ingest for every tenant at once. A store without a byte cap now drops its oldest day once its filesystem is 80% full. VictoriaTraces refuses to start with both limits set, so a byte cap, when given, replaces the percentage one. It always keeps the last two days, which the values text now says. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
This commit is contained in:
Alexey Artamonov committed
1 parent
2d10565873
commit
d5fc723208
6 files changed
+91
-15
No files matched your search
@@ -55,16 +55,16 @@
|
||||
|
||||
### Traces storage configuration
|
||||
|
||||
| Name | Description | Type | Value |
|
||||
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------ |
|
||||
| `tracingStorages` | Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed. | `[]object` | `[]` |
|
||||
| `tracingStorages[i].name` | Name of the storage instance: a lowercase RFC 1123 label of at most 42 characters. Treat as immutable once a backend exists: the VTCluster/VTSingle, its Services, PVCs and Grafana datasource are all named after it, so renaming an entry deploys a new, empty backend and removes the old one, the same way changing mode does (single mode loses the old PVC, cluster mode leaves the old vtstorage PVCs Bound until the whole Monitoring release is deleted). | `string` | `""` |
|
||||
| `tracingStorages[i].mode` | Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters. Treat as immutable once a backend exists: mode selects which Kind is rendered under the same name, so changing it on a running backend re-identifies the object rather than reconfiguring it. Switching from "single" to "cluster" prunes the VTSingle and the operator garbage-collects its PVC (stored traces are lost); switching from "cluster" to "single" (or removing the entry) leaves the vtstorage PVCs Bound and billed until the whole Monitoring release is deleted. The API cannot guard this (transition rules are not available on an array-item field), so this note is the only guard: migrate deliberately, accepting the loss. | `string` | `cluster` |
|
||||
| `tracingStorages[i].retentionPeriod` | Retention period for traces. | `string` | `14d` |
|
||||
| `tracingStorages[i].storage` | Persistent volume size. | `quantity` | `10Gi` |
|
||||
| `tracingStorages[i].storageClassName` | StorageClass used to store the data. | `string` | `replicated` |
|
||||
| `tracingStorages[i].retentionDiskUsageBytes` | Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. | `string` | `""` |
|
||||
| `tracingCentralHost` | tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: with retentionDiskUsageBytes set the oldest spans of all of them are evicted together, without it the disk fills for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts. | `bool` | `false` |
|
||||
| Name | Description | Type | Value |
|
||||
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------ |
|
||||
| `tracingStorages` | Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed. | `[]object` | `[]` |
|
||||
| `tracingStorages[i].name` | Name of the storage instance: a lowercase RFC 1123 label of at most 42 characters. Treat as immutable once a backend exists: the VTCluster/VTSingle, its Services, PVCs and Grafana datasource are all named after it, so renaming an entry deploys a new, empty backend and removes the old one, the same way changing mode does (single mode loses the old PVC, cluster mode leaves the old vtstorage PVCs Bound until the whole Monitoring release is deleted). | `string` | `""` |
|
||||
| `tracingStorages[i].mode` | Deployment mode: "cluster" renders a VTCluster; "single" renders a single-binary VTSingle for edge/dev/small clusters. Treat as immutable once a backend exists: mode selects which Kind is rendered under the same name, so changing it on a running backend re-identifies the object rather than reconfiguring it. Switching from "single" to "cluster" prunes the VTSingle and the operator garbage-collects its PVC (stored traces are lost); switching from "cluster" to "single" (or removing the entry) leaves the vtstorage PVCs Bound and billed until the whole Monitoring release is deleted. The API cannot guard this (transition rules are not available on an array-item field), so this note is the only guard: migrate deliberately, accepting the loss. | `string` | `cluster` |
|
||||
| `tracingStorages[i].retentionPeriod` | Retention period for traces. | `string` | `14d` |
|
||||
| `tracingStorages[i].storage` | Persistent volume size. | `quantity` | `10Gi` |
|
||||
| `tracingStorages[i].storageClassName` | StorageClass used to store the data. | `string` | `replicated` |
|
||||
| `tracingStorages[i].retentionDiskUsageBytes` | Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. Unset, the store drops its oldest day once its filesystem is 80% full instead, so every store is disk-bounded. Either way VictoriaTraces keeps the last two days, so spans arriving faster than the cap allows for two days can still fill the disk. | `string` | `""` |
|
||||
| `tracingCentralHost` | tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts. | `bool` | `false` |
|
||||
|
||||
|
||||
### Traces collector configuration
|
||||
|
||||
@@ -351,7 +351,7 @@
|
||||
"type": "string"
|
||||
},
|
||||
"retentionDiskUsageBytes": {
|
||||
"description": "Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. \"8GB\" or \"8GiB\"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so \"8Gi\" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value.",
|
||||
"description": "Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. \"8GB\" or \"8GiB\"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so \"8Gi\" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. Unset, the store drops its oldest day once its filesystem is 80% full instead, so every store is disk-bounded. Either way VictoriaTraces keeps the last two days, so spans arriving faster than the cap allows for two days can still fill the disk.",
|
||||
"type": "string"
|
||||
},
|
||||
"retentionPeriod": {
|
||||
@@ -382,7 +382,7 @@
|
||||
}
|
||||
},
|
||||
"tracingCentralHost": {
|
||||
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: with retentionDiskUsageBytes set the oldest spans of all of them are evicted together, without it the disk fills for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
|
||||
"description": "tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.",
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
|
||||
@@ -106,12 +106,12 @@ logsStorages:
|
||||
## @field {string} retentionPeriod="14d" - Retention period for traces.
|
||||
## @field {quantity} storage="10Gi" - Persistent volume size.
|
||||
## @field {string} storageClassName="replicated" - StorageClass used to store the data.
|
||||
## @field {string} [retentionDiskUsageBytes] - Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value.
|
||||
## @field {string} [retentionDiskUsageBytes] - Optional disk-usage cap as a VictoriaMetrics BytesString (e.g. "8GB" or "8GiB"), bounding disk use on top of the age-based retentionPeriod. Uses byte-suffix grammar (kb/mb/gb/tb, KiB/MiB/GiB/TiB), NOT the Kubernetes-quantity grammar of the sibling storage field, so "8Gi" is rejected by the operator. In cluster mode the cap is per vtstorage replica (replicaCount 2), so the effective total is twice the value. Unset, the store drops its oldest day once its filesystem is 80% full instead, so every store is disk-bounded. Either way VictoriaTraces keeps the last two days, so spans arriving faster than the cap allows for two days can still fill the disk.
|
||||
|
||||
## @param {[]TracingStorage} tracingStorages - Configuration of traces storage instances. Empty by default (tracing off): no backend and no trace collector are rendered. Adding an entry provisions a VictoriaTraces backend and, while tracingCollector.enabled is true, deploys the OTLP collector in front of it. A tenant with shared-central tracing (the Tenant's tracingCentral) gets the collector without an entry, which then exports to the shared store; entries still listed keep their store and datasource, so traces stored before the switch stay readable until the entry is removed.
|
||||
tracingStorages: []
|
||||
|
||||
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space per tenant: with retentionDiskUsageBytes set the oldest spans of all of them are evicted together, without it the disk fills for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
|
||||
## @param {bool} tracingCentralHost - tenant-root only: accept spans from tenants that turned on shared-central tracing (the Tenant's tracingCentral) into this Monitoring's tracingStorages entry named `generic`, which must be in `cluster` mode. Every central tenant then shares that store's disk and retention with tenant-root, and nothing reserves space or limits the rate per tenant: once the store's disk cap is reached (retentionDiskUsageBytes, or 80% of its filesystem) the oldest spans of all of them are evicted together, and a tenant sending faster than the cap allows for the two days VictoriaTraces always keeps can fill the disk for all of them. Ignored outside tenant-root. The tenant-root namespace is inside the trust boundary of the shared store, as it is for the metrics of tenants that rely on a parent's monitoring: vmauth pins each central tenant to its own account, but workloads in tenant-root reach the store directly, and one that learns a tenant's account ID (a hash of its namespace name and UID, which its VMUser carries) can read and write that account. Whoever tenant-root admits to its Grafana is inside it too, since tenant-root's own datasource for the store reaches vtselect without vmauth; who that is follows its Grafana's oidc mode (with System, tenant-root's own groups, which already hold the same access level in every tenant under tenant-root). The render refuses to stop hosting while central tenants remain, but removing the tenant-root Monitoring altogether is not a render and is not refused: turn tracingCentral off on the central tenants first, since until then every Monitoring render of theirs fails, their metrics, logs and alerting changes included. The refusals read each other's objects when they render, so a tenant turning tracingCentral on while tenant-root stops hosting can pass both: that tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning that tenant's tracingCentral off instead ends the silent drop, and turning it on again is then refused until tenant-root hosts.
|
||||
tracingCentralHost: false
|
||||
|
||||
##
|
||||
|
||||
+1
-1
@@ -20,7 +20,7 @@ spec:
|
||||
singular: monitoring
|
||||
plural: monitorings
|
||||
openAPISchema: |-
|
||||
{"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access Grafana externally (defaults to 'grafana' subdomain for the tenant host).","type":"string","default":""},"metricsStorages":{"description":"Configuration of metrics storage instances.","type":"array","default":[{"deduplicationInterval":"15s","name":"shortterm","retentionPeriod":"3d","storage":"10Gi","storageClassName":""},{"deduplicationInterval":"5m","name":"longterm","retentionPeriod":"14d","storage":"10Gi","storageClassName":""}],"items":{"type":"object","required":["deduplicationInterval","name","retentionPeriod","storage"],"properties":{"deduplicationInterval":{"description":"Deduplication interval for metrics.","type":"string"},"name":{"description":"Name of the storage instance.","type":"string"},"retentionPeriod":{"description":"Retention period for metrics.","type":"string"},"storage":{"description":"Persistent volume size.","type":"string","default":"10Gi"},"storageClassName":{"description":"StorageClass used for the data.","type":"string"},"vminsert":{"description":"Configuration for vminsert.","type":"object","properties":{"maxAllowed":{"description":"Maximum allowed resources.","type":"object","properties":{"cpu":{"description":"CPU limit.","default":1,"pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory limit.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"minAllowed":{"description":"Minimum guaranteed resources.","type":"object","properties":{"cpu":{"description":"CPU request.","default":"100m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory request.","default":"256Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}}}},"vmselect":{"description":"Configuration for vmselect.","type":"object","properties":{"maxAllowed":{"description":"Maximum allowed resources.","type":"object","properties":{"cpu":{"description":"CPU limit.","default":1,"pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory limit.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"minAllowed":{"description":"Minimum guaranteed resources.","type":"object","properties":{"cpu":{"description":"CPU request.","default":"100m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory request.","default":"256Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}}}},"vmstorage":{"description":"Configuration for vmstorage.","type":"object","properties":{"maxAllowed":{"description":"Maximum allowed resources.","type":"object","properties":{"cpu":{"description":"CPU limit.","default":1,"pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory limit.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"minAllowed":{"description":"Minimum guaranteed resources.","type":"object","properties":{"cpu":{"description":"CPU request.","default":"100m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory request.","default":"256Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0Line truncated
|
||||
{"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access Grafana externally (defaults to 'grafana' subdomain for the tenant host).","type":"string","default":""},"metricsStorages":{"description":"Configuration of metrics storage instances.","type":"array","default":[{"deduplicationInterval":"15s","name":"shortterm","retentionPeriod":"3d","storage":"10Gi","storageClassName":""},{"deduplicationInterval":"5m","name":"longterm","retentionPeriod":"14d","storage":"10Gi","storageClassName":""}],"items":{"type":"object","required":["deduplicationInterval","name","retentionPeriod","storage"],"properties":{"deduplicationInterval":{"description":"Deduplication interval for metrics.","type":"string"},"name":{"description":"Name of the storage instance.","type":"string"},"retentionPeriod":{"description":"Retention period for metrics.","type":"string"},"storage":{"description":"Persistent volume size.","type":"string","default":"10Gi"},"storageClassName":{"description":"StorageClass used for the data.","type":"string"},"vminsert":{"description":"Configuration for vminsert.","type":"object","properties":{"maxAllowed":{"description":"Maximum allowed resources.","type":"object","properties":{"cpu":{"description":"CPU limit.","default":1,"pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory limit.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"minAllowed":{"description":"Minimum guaranteed resources.","type":"object","properties":{"cpu":{"description":"CPU request.","default":"100m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory request.","default":"256Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}}}},"vmselect":{"description":"Configuration for vmselect.","type":"object","properties":{"maxAllowed":{"description":"Maximum allowed resources.","type":"object","properties":{"cpu":{"description":"CPU limit.","default":1,"pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory limit.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"minAllowed":{"description":"Minimum guaranteed resources.","type":"object","properties":{"cpu":{"description":"CPU request.","default":"100m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory request.","default":"256Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}}}},"vmstorage":{"description":"Configuration for vmstorage.","type":"object","properties":{"maxAllowed":{"description":"Maximum allowed resources.","type":"object","properties":{"cpu":{"description":"CPU limit.","default":1,"pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory limit.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"minAllowed":{"description":"Minimum guaranteed resources.","type":"object","properties":{"cpu":{"description":"CPU request.","default":"100m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory request.","default":"256Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0Line truncated
|
||||
release:
|
||||
prefix: ""
|
||||
labels:
|
||||
|
||||
@@ -12,6 +12,12 @@
|
||||
export. The entries still listed here stay rendered: the switch belongs to the
|
||||
parent tenant and the stores to this one, and a VTSingle takes its PVC with it.
|
||||
|
||||
Every store is disk-bounded, not only age-bounded, so a full PVC cannot
|
||||
block ingest: without a byte cap it drops its oldest day once its filesystem
|
||||
is 80% full. VictoriaTraces refuses to start with both limits, so only one is
|
||||
rendered. It always keeps the last two days, which a flood faster than that
|
||||
can still push past the limit.
|
||||
|
||||
The readiness-gate healthCheckExprs on VTCluster/VTSingle stay unconditional:
|
||||
Flux evaluates them only against CRs present in the release inventory, so with
|
||||
nothing rendered they are inert.
|
||||
@@ -34,6 +40,9 @@ spec:
|
||||
retentionPeriod: {{ .retentionPeriod | quote }}
|
||||
{{- with .retentionDiskUsageBytes }}
|
||||
retentionMaxDiskSpaceUsageBytes: {{ . | quote }}
|
||||
{{- else }}
|
||||
extraArgs:
|
||||
retention.maxDiskUsagePercent: "80"
|
||||
{{- end }}
|
||||
# Stamped on the storage PVC so the monitoring post-delete cleanup hook can
|
||||
# select it by this label (same release-scoped cleanup as logs/metrics).
|
||||
@@ -70,6 +79,9 @@ spec:
|
||||
retentionPeriod: {{ .retentionPeriod | quote }}
|
||||
{{- with .retentionDiskUsageBytes }}
|
||||
retentionMaxDiskSpaceUsageBytes: {{ . | quote }}
|
||||
{{- else }}
|
||||
extraArgs:
|
||||
retention.maxDiskUsagePercent: "80"
|
||||
{{- end }}
|
||||
replicaCount: 2
|
||||
resources: {}
|
||||
|
||||
@@ -123,6 +123,70 @@ tests:
|
||||
path: spec.retentionMaxDiskSpaceUsageBytes
|
||||
value: "4GB"
|
||||
|
||||
# The design requires every traces backend to be disk-bounded, not only
|
||||
# age-bounded, so a full PVC cannot block ingest. Without an explicit byte cap
|
||||
# the store drops its oldest day once its filesystem is 80% full. The two
|
||||
# flags are mutually exclusive: VictoriaTraces refuses to start with both.
|
||||
- it: bounds a cluster-mode store at 80% of its filesystem without a byte cap
|
||||
templates:
|
||||
- templates/vtraces/vtraces.yaml
|
||||
set:
|
||||
tracingStorages:
|
||||
- name: generic
|
||||
mode: cluster
|
||||
retentionPeriod: "14d"
|
||||
storage: 10Gi
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.storage.extraArgs["retention.maxDiskUsagePercent"]
|
||||
value: "80"
|
||||
- notExists:
|
||||
path: spec.storage.retentionMaxDiskSpaceUsageBytes
|
||||
|
||||
- it: bounds a single-mode store at 80% of its filesystem without a byte cap
|
||||
templates:
|
||||
- templates/vtraces/vtraces.yaml
|
||||
set:
|
||||
tracingStorages:
|
||||
- name: generic
|
||||
mode: single
|
||||
retentionPeriod: "14d"
|
||||
storage: 10Gi
|
||||
asserts:
|
||||
- equal:
|
||||
path: spec.extraArgs["retention.maxDiskUsagePercent"]
|
||||
value: "80"
|
||||
- notExists:
|
||||
path: spec.retentionMaxDiskSpaceUsageBytes
|
||||
|
||||
- it: sets no percentage cap on a cluster-mode store that has a byte cap
|
||||
templates:
|
||||
- templates/vtraces/vtraces.yaml
|
||||
set:
|
||||
tracingStorages:
|
||||
- name: generic
|
||||
mode: cluster
|
||||
retentionPeriod: "14d"
|
||||
storage: 10Gi
|
||||
retentionDiskUsageBytes: "8GB"
|
||||
asserts:
|
||||
- notExists:
|
||||
path: spec.storage.extraArgs
|
||||
|
||||
- it: sets no percentage cap on a single-mode store that has a byte cap
|
||||
templates:
|
||||
- templates/vtraces/vtraces.yaml
|
||||
set:
|
||||
tracingStorages:
|
||||
- name: generic
|
||||
mode: single
|
||||
retentionPeriod: "14d"
|
||||
storage: 10Gi
|
||||
retentionDiskUsageBytes: "4GB"
|
||||
asserts:
|
||||
- notExists:
|
||||
path: spec.extraArgs
|
||||
|
||||
- it: renders a Jaeger-typed Grafana datasource at vtselect:10471 for cluster mode
|
||||
templates:
|
||||
- templates/vtraces/grafana-datasource.yaml
|
||||
|
||||
Reference in new issue
Block a user