Files
cozystack/packages/extra/monitoring/tests/helmrelease_test.yaml
T
Alexey Artamonov f9afc73096 fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.

Three complementary changes:

- Readiness (do not stay silently Ready): the monitoring-system
  HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
  its VLCluster (status.updateStatus), so a missing/unhealthy log store
  makes the release not-ready instead of flipping Ready as soon as the
  CR is applied. Mirrors the seaweedfs CNPG pattern.

- No empty-log-store hole: system/monitoring rejects an empty
  logsStorages at render time (helm fail). With no entry no VLCluster is
  created, the readiness gate has nothing to match, and fluent-bit would
  silently drop logs — so this is made a loud misconfiguration rather
  than a documented footgun.

- No silent fallback: the cozy-monitoring -> tenant-root
  vlinsert/vminsert ExternalName redirects in cozystack-basics are now
  created only when the root tenant hosts monitoring, gated on
  _cluster.monitoring-enabled — fed from a new platform value
  monitoring.rootEnabled (default true, preserving the historical
  always-create behaviour). monitoring.rootEnabled is the single
  render-time source of truth: changing it re-renders cozystack-basics
  (helm-controller watches the cozystack-values Secret it flows into). A
  live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
  cozystack-basics is an operator-generated HelmRelease with drift
  detection off, so it re-renders only on install/upgrade/values-change
  — never on a reconcile tick or when a lookup result changes — and a
  lookup would go stale in both directions. Operators must keep
  monitoring.rootEnabled in sync with Tenant/root spec.monitoring.

Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.

Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-07-08 22:27:22 +03:00

45 lines
1.6 KiB
YAML

suite: monitoring-system HelmRelease readiness gate
templates:
- templates/helmrelease.yaml
release:
name: monitoring
namespace: tenant-root
# The monitoring-system HelmRelease owns the VLCluster(s) rendered from
# logsStorages. Without a health gate it flips Ready as soon as helm applies the
# CR, so a missing or broken VictoriaLogs stays invisible while fluent-bit drops
# logs at a dead-end vlinsert-generic (issue #3181). A `poller` waitStrategy plus
# a VLCluster healthCheckExpr makes that failure surface as not-ready.
tests:
- it: names the child release <release>-system
asserts:
- containsDocument:
kind: HelmRelease
apiVersion: helm.toolkit.fluxcd.io/v2
name: monitoring-system
- it: uses the poller waitStrategy so healthCheckExprs are evaluated
asserts:
- equal:
path: spec.waitStrategy.name
value: poller
- it: gates readiness on the VLCluster status.updateStatus
asserts:
- equal:
path: spec.healthCheckExprs[0].apiVersion
value: operator.victoriametrics.com/v1
- equal:
path: spec.healthCheckExprs[0].kind
value: VLCluster
- equal:
path: spec.healthCheckExprs[0].current
value: "has(status.updateStatus) && status.updateStatus == 'operational'"
- equal:
path: spec.healthCheckExprs[0].inProgress
value: "has(status.updateStatus) && status.updateStatus == 'expanding'"
- equal:
path: spec.healthCheckExprs[0].failed
value: "has(status.updateStatus) && status.updateStatus != 'operational' && status.updateStatus != 'expanding'"