mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
82 lines
2.6 KiB
YAML
82 lines
2.6 KiB
YAML
suite: monitoring ExternalName redirects in cozy-monitoring
|
|
templates:
|
|
- templates/monitoring-external-services.yaml
|
|
|
|
release:
|
|
name: cozystack-basics
|
|
namespace: cozy-system
|
|
|
|
# The vlinsert-generic / vminsert-shortterm / vminsert-longterm ExternalName
|
|
# Services redirect cozy-monitoring at the platform monitoring stack the root
|
|
# tenant hosts in tenant-root. They only resolve when the root tenant actually
|
|
# runs that stack (Tenant/root spec.monitoring=true). When it is off the
|
|
# redirects dead-end and fluent-bit silently drops logs (issue #3181), so they
|
|
# must NOT be rendered.
|
|
#
|
|
# The render is gated solely on _cluster.monitoring-enabled (fed from the
|
|
# platform monitoring.rootEnabled value; operators keep it in sync with
|
|
# Tenant/root spec.monitoring). The "true" default preserves the historical
|
|
# always-create behaviour for cozystack-values Secrets written before the key
|
|
# existed, and a null _cluster must not panic. The cases below pin the full
|
|
# contract: enabled, disabled, default, and the nil guard.
|
|
tests:
|
|
- it: renders all three ExternalName redirects when monitoring-enabled is "true"
|
|
set:
|
|
_cluster:
|
|
monitoring-enabled: "true"
|
|
asserts:
|
|
- hasDocuments:
|
|
count: 3
|
|
- equal:
|
|
path: metadata.name
|
|
value: vlinsert-generic
|
|
documentIndex: 0
|
|
- equal:
|
|
path: metadata.namespace
|
|
value: cozy-monitoring
|
|
documentIndex: 0
|
|
- equal:
|
|
path: metadata.name
|
|
value: vminsert-shortterm
|
|
documentIndex: 1
|
|
- equal:
|
|
path: metadata.name
|
|
value: vminsert-longterm
|
|
documentIndex: 2
|
|
|
|
- it: keeps the redirects (default) when monitoring-enabled is absent
|
|
asserts:
|
|
- hasDocuments:
|
|
count: 3
|
|
|
|
- it: omits all redirects when monitoring-enabled is "false"
|
|
set:
|
|
_cluster:
|
|
monitoring-enabled: "false"
|
|
asserts:
|
|
- hasDocuments:
|
|
count: 0
|
|
|
|
# Guard: a null _cluster (missing cozystack-values Secret / dry-run) must not
|
|
# panic the render; `index (.Values._cluster | default dict) ...` falls back
|
|
# to the "true" default and keeps the redirects.
|
|
- it: keeps the redirects (no panic) when _cluster is null
|
|
set:
|
|
_cluster: null
|
|
asserts:
|
|
- hasDocuments:
|
|
count: 3
|
|
|
|
- it: vlinsert-generic still points at tenant-root when rendered
|
|
set:
|
|
_cluster:
|
|
monitoring-enabled: "true"
|
|
documentIndex: 0
|
|
asserts:
|
|
- equal:
|
|
path: spec.type
|
|
value: ExternalName
|
|
- equal:
|
|
path: spec.externalName
|
|
value: vlinsert-generic.tenant-root.svc.cluster.local
|