mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 04:33:36 +09:00
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
45 lines
1.6 KiB
YAML
45 lines
1.6 KiB
YAML
suite: monitoring-system HelmRelease readiness gate
|
|
templates:
|
|
- templates/helmrelease.yaml
|
|
|
|
release:
|
|
name: monitoring
|
|
namespace: tenant-root
|
|
|
|
# The monitoring-system HelmRelease owns the VLCluster(s) rendered from
|
|
# logsStorages. Without a health gate it flips Ready as soon as helm applies the
|
|
# CR, so a missing or broken VictoriaLogs stays invisible while fluent-bit drops
|
|
# logs at a dead-end vlinsert-generic (issue #3181). A `poller` waitStrategy plus
|
|
# a VLCluster healthCheckExpr makes that failure surface as not-ready.
|
|
tests:
|
|
- it: names the child release <release>-system
|
|
asserts:
|
|
- containsDocument:
|
|
kind: HelmRelease
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
name: monitoring-system
|
|
|
|
- it: uses the poller waitStrategy so healthCheckExprs are evaluated
|
|
asserts:
|
|
- equal:
|
|
path: spec.waitStrategy.name
|
|
value: poller
|
|
|
|
- it: gates readiness on the VLCluster status.updateStatus
|
|
asserts:
|
|
- equal:
|
|
path: spec.healthCheckExprs[0].apiVersion
|
|
value: operator.victoriametrics.com/v1
|
|
- equal:
|
|
path: spec.healthCheckExprs[0].kind
|
|
value: VLCluster
|
|
- equal:
|
|
path: spec.healthCheckExprs[0].current
|
|
value: "has(status.updateStatus) && status.updateStatus == 'operational'"
|
|
- equal:
|
|
path: spec.healthCheckExprs[0].inProgress
|
|
value: "has(status.updateStatus) && status.updateStatus == 'expanding'"
|
|
- equal:
|
|
path: spec.healthCheckExprs[0].failed
|
|
value: "has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'"
|