A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.
Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Each collector pod starts with a full token bucket, and burst was at
least 64 MiB, the largest request vtinsert accepts. A tenant controls
when its collector pods start, through any Monitoring change or by
deleting them, and how many there are, so it could write tens of
mebibytes above its rate per pod start and gigabytes per rollout of
many replicas, which the two-day bound does not survive.
The limiter only sees requests the receiver admits, so in central mode
the receiver now caps them at 4 MiB, decompressed for HTTP, and burst
drops to that floor; a pod start lets at most 4 MiB through above the
rate. The replica count is capped at 10. The texts now say what the
default rate guarantees: one tenant writes at most a default store's
disk cap over two days counted uncompressed, rather than that it
cannot fill it, and they name the per-pod-start allowance and that a
sender pinned to one replica gets that replica's share.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.
The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.
Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The apps API defaults a Monitoring spec from its schema but does not
prune keys the schema leaves out, and the Monitoring chart hands its
values to the inner chart as they are. A tenant could therefore set
tracingCollector.image and run any collector it liked in its
namespace; with a per-tenant rate limit in that collector, the same
key would let a tenant drop the limit. The image is the platform's,
so it is no longer passed through.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A per-tenant traces backend costs every tenant its own VictoriaTraces
stack. A tenant can now send new spans to one shared store in
tenant-root instead, switched on by _namespace.tracingCentral, which
the tenant chart publishes and the apps API does not let an
application set. tenant-root decides whether it hosts that store, with
tracingCentralHost next to a cluster-mode store named generic.
VictoriaTraces takes the tenant account from request headers and does
no authorization of its own, so other tenants reach the store only
through a vmauth in tenant-root; tenant-root itself, with whoever it
admits to its Grafana, stays inside the trust boundary. Each central
tenant gets a VMUser that pins an account derived from its namespace
name and UID for both writes and reads and caps its requests in
flight, which the collector's two export senders stay under; its
collector exports and its Grafana reads with credentials the chart
generates and keeps, and vmauth overrides any account a client sends.
The chart owns those credentials because the operator publishes a
VMUser's own Secret only once a VMAuth selects it; the collector rolls
when they change, since it reads them at start. The vmauth internal
routes, whose /metrics names every tenant's user, are served on a port
tenants cannot reach, and the vmauth is sized by a VPA, spread across
nodes under a disruption budget, has a WorkloadMonitor like the other
monitoring workloads, and may reach the apiserver, where its
config-reloader reads its config Secret.
Renders refuse the states in which spans would be dropped with every
object Ready: a central tenant while tenant-root hosts nothing,
tenant-root dropping the hosting while central tenants remain, and
the host flag without a store to host. Each refusal reads the other
side at render time, so a concurrent opt-in and stop-hosting can pass
both; the values text names that race and its recovery. At run time
tenant-root's release is not Ready while the vmauth has failed.
Only the collector's export moves: local stores the tenant still lists
keep rendering with their datasources, because the switch belongs to
the parent tenant and the stores to this one, and the one-store limit
of the collector holds in both modes, since turning central off makes
it export locally again. tenant-root never counts as central.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The Role listed the WorkloadMonitor as 'alermanager', so the real
'alertmanager' monitor was never granted: a tenant dashboard is
forbidden from reading its status and the alertmanager health tile
breaks. The test pins the correct name and the absence of the typo.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.
The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.
redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.
The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The timeout, waitStrategy and healthCheckExprs rationale in the
monitoring-system HelmRelease was written as `#` comments, which Helm
renders verbatim, so every tenant's HelmRelease carried about thirty lines
of prose. Move it into template comments like the traces paragraph beside
it: the reasoning stays in the chart source and the rendered object is
unchanged apart from losing the comments.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The `#` prose in the traces datasource and in the HelmRelease health-check
block is not stripped at render the way `{{/* */}}` is, so it shipped into
every tenant's rendered GrafanaDatasource and HelmRelease. Move it into
template comments: the reasoning stays in the chart source but no longer lands
in the emitted objects.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The healthCheckExprs comment (and the mirroring test comment) still claimed
tracingStorages is render-rejected when empty and that the 'Ready with no
trace store' hole cannot be reached, but defaulting tracing off removed that
guard and made empty the default. Rewrite both to state the actual behaviour:
empty is the default (tracing off), so the exprs are usually inert (Flux
evaluates them only against CRs present in the inventory), and a Ready release
with no trace store is intentional and harmless because nothing produces
spans without a collector, so no fail-on-empty guard is needed.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The chart previously injected the two settings the users-Job's
contract depends on — `skip_org_role_sync = true` and
`oauth_allow_insecure_email_lookup = true` — only inside the
`mode: System` branch of `grafana.yaml`. `values.yaml` and
`docs/oidc-grafana.md` both promised the settings would be forced
regardless of mode, but under `mode: CustomConfig` the operator's
inline `auth.generic_oauth` map was written verbatim, both
settings were absent, and the users-Job's role assignments were
silently stranded (no lookup by email → new Grafana account minted
at login) or transiently overwritten (no skip_org_role_sync → any
`role_attribute_path` or Grafana's `auto_assign_org_role: Viewer`
overwrote the Job's role on each login).
Fix: merge a `$chartForcedOidc` dict on top of both the generated
System block and the operator's inline map (`merge` wins for
overlapping keys, so the operator can NOT override the two
settings). CustomConfig secretRef mode cannot receive the merge
because the ini fragment is authoritative — the chart now hard-fails
render when both `customConfig.secretRef.name` and non-empty
`spec.oidc.users` are set, forcing the operator to either switch to
inline config or manage authorization themselves inside the ini.
Also:
- Align the inner HR `timeout` on `packages/extra/monitoring/templates/helmrelease.yaml`
from 10m to 15m so it matches the outer cozyrds
`release.cozystack.io/helm-install-timeout: "15m"` and does not
fail while the users-Job is still inside its
`activeDeadlineSeconds: 900`.
- Route the users-Job image through `cozy-lib.image` so
mirrored / air-gapped installations resolve it (matches the
pattern in `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`).
- Add `spec.ttlSecondsAfterFinished: 3600` on the users-Job so a
succeeded Job + Pod are garbage-collected on the next reconcile
interval rather than lingering until the next hook activation
(which never fires on release uninstall).
- Harden the users-Job shell:
* emails read NUL-terminated (`jq --raw-output0`) so an address
containing whitespace cannot corrupt the `for email in ...`
loop's field splitting;
* `/api/users/lookup?loginOrEmail=<email>` URL-encoded via
`jq @uri` so `&`/`=`/`?` in the local part parse correctly on
the Grafana side;
* prune loop switched from pipe-delimited fields to TSV inside
NUL-terminated records, with `cut -f` for extraction.
- Add a template-level assertion (`monitoring.oidc.assertUsersEmailShape`)
rejecting emails that do not match a conservative regex —
cheaper failure mode than a 400 from Grafana.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
## What this PR does
Addresses #3181, where platform log ingestion could fail **silently**:
fluent-bit shipped to `vlinsert-generic` while the monitoring
`HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName`
in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did
not exist when the root tenant had monitoring disabled. Logs were
dropped with no failing signal.
Three complementary layers make the failure loud instead of silent:
1. **Readiness gate** —
`packages/extra/monitoring/templates/helmrelease.yaml` now sets a
`poller` `waitStrategy` and a `healthCheckExprs` entry on the
`VLCluster` `status.updateStatus`. A missing/unhealthy log store makes
the `monitoring-system` release **not-ready** instead of flipping
`Ready` the moment the CR is applied. Mirrors the existing seaweedfs
CNPG pattern.
2. **No empty-log-store hole** —
`packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render
on an empty `logsStorages`. With no entry no `VLCluster` exists, the
readiness gate has nothing to match, and fluent-bit would silently drop
logs — so this is now a loud misconfiguration.
3. **No silent fallback** — the `cozy-monitoring → tenant-root`
`vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are
created only when the root tenant hosts monitoring. The template reads
the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth,
so it cannot drift from the tenant's real state) and falls back to a new
platform value `monitoring.rootEnabled` (via
`_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first
render). The default preserves the historical always-create behaviour.
Covered by helm-unittest across all four affected charts (141 tests):
readiness-gate fields, empty-vs-populated `logsStorages` contract,
`ExternalName` present/absent fallback contract, and the platform wiring
into the `cozystack-values` Secret. Also adds the missing `test:` target
to `packages/extra/monitoring/Makefile` so its suite runs in CI.
**Known limitation / follow-up:** the legacy platform monitoring release
installed directly into `cozy-monitoring` via the `cozystack.monitoring`
PackageSource cannot carry the readiness gate, because the engine's
`ComponentInstall`/HelmRelease builder and the vendored
`helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`.
That source is not referenced by any active bundle; full coverage there
needs a helm-controller bump plus engine support.
### Release note
```release-note
fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a platform setting (`monitoring.rootEnabled`) to control whether
root-level monitoring is enabled.
* Monitoring redirect ExternalName services are now rendered only when
root monitoring is enabled (defaulting to enabled).
* **Bug Fixes**
* Ensures an explicitly set `monitoring.rootEnabled=false` is preserved
when generating platform configuration.
* Prevents incomplete monitoring deployments by failing rendering when
`logsStorages` is empty.
* Improves VictoriaMetrics readiness by delaying “ready” until the
operator reports an operational status.
* **Tests**
* Added/expanded Helm-unittest suites covering monitoring wiring,
redirects, readiness gating, and `logsStorages` requirements.
* Added a `make test` target to run chart unittests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Re-review on #3094 (IvanHunters): the AND selector kept managed-by=vm-operator,
which on the shipped vm-operator v0.68.4 is not reliably present on the storage
PVCs (it lives on the StatefulSet selector/pod labels, not the claim template),
making the delete fragile. Match by apps.cozystack.io/application.name=<release>-system
alone: that label is stamped on the VM/VL claim templates (67b087ab) and copied
onto the PVCs by IntoSTSVolume even on v0.68.4, is already release-scoped, and is
absent from the CNPG db PVCs (grafana-db/alerta-db carry only policy.cozystack.io
+ cnpg.io/*), so those stay untouched. Regression guard now forbids re-adding
managed-by to the selector.
Also apply the review's discipline items: the cleanup Job keeps only
before-hook-creation (a Failed run survives for post-mortem; hook-succeeded stays
on SA/Role/RoleBinding, cf. #3072) and sets backoffLimit: 0 (best-effort, no retry).
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added
in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The
apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster
spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto
PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091
storage leak stayed open.
Stamp the label where it actually reaches the PVCs: on
spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels
of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies
claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the
existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name=
<release>-system) now matches and stays scoped to this release. Verified by
rendering the CRs (label resolves to monitoring-system, matching the hook).
Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei
non-blocking) and a helm-unittest suite asserting the label lands on all three
storage claim templates. Corrected the misleading test comment about how the
label reaches the PVCs.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Address IvanHunters review on #3094: the PVC delete selector was
release-agnostic (-l managed-by=vm-operator), matching the VM storage
PVCs of every VMCluster/VLCluster in the namespace. If a namespace ever
hosted more than one monitoring release, tearing one down would delete
the other's VictoriaMetrics/VictoriaLogs storage.
Scope the selector to this release via apps.cozystack.io/application.name=
<release>-system, the ownership label the monitoring chart stamps on the
VM CRs' managedMetadata and vm-operator propagates onto the PVCs — mirroring
the merged seaweedfs hook's release scoping. deletecollection is still not
needed: kubectl delete -l lists then deletes individually.
Tests: assert the release-scoped selector and add a regression guard that
the delete never falls back to the bare managed-by=vm-operator selector.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Address review on #3094:
- Add --wait=false to both kubectl delete calls so PVC finalizers
(pvc-protection on pods still terminating) can never block the
post-delete hook and stall the HelmRelease teardown.
- Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop.
- Tests: assert the non-blocking flag, plus negative assertions that the
PVC delete stays label-scoped (no --all) and the secret delete stays
name-scoped (no -l/--all).
deletecollection RBAC verb intentionally not added: kubectl delete -l
lists then deletes individually, so the existing delete verb suffices.
BREAKING CHANGE: disabling monitoring now permanently removes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
When the monitoring tenant module is disabled, the vm-operator storage PVCs
(VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left
orphaned. Add a post-delete cleanup hook that deletes the PVCs by the
managed-by=vm-operator label and the two TLS secrets by name, plus a test
target and helm-unittest suite (the package had neither). CNPG PVCs are not
targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only
root fs, dropped caps, resource limits); failures are logged, never blocking
teardown.
NOTE: disabling monitoring now permanently removes its storage PVCs and the
data they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.
Three complementary changes:
- Readiness (do not stay silently Ready): the monitoring-system
HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
its VLCluster (status.updateStatus), so a missing/unhealthy log store
makes the release not-ready instead of flipping Ready as soon as the
CR is applied. Mirrors the seaweedfs CNPG pattern.
- No empty-log-store hole: system/monitoring rejects an empty
logsStorages at render time (helm fail). With no entry no VLCluster is
created, the readiness gate has nothing to match, and fluent-bit would
silently drop logs — so this is made a loud misconfiguration rather
than a documented footgun.
- No silent fallback: the cozy-monitoring -> tenant-root
vlinsert/vminsert ExternalName redirects in cozystack-basics are now
created only when the root tenant hosts monitoring, gated on
_cluster.monitoring-enabled — fed from a new platform value
monitoring.rootEnabled (default true, preserving the historical
always-create behaviour). monitoring.rootEnabled is the single
render-time source of truth: changing it re-renders cozystack-basics
(helm-controller watches the cozystack-values Secret it flows into). A
live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
cozystack-basics is an operator-generated HelmRelease with drift
detection off, so it re-renders only on install/upgrade/values-change
— never on a reconcile tick or when a lookup result changes — and a
lookup would go stale in both directions. Operators must keep
monitoring.rootEnabled in sync with Tenant/root spec.monitoring.
Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.
Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.
Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.
Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7e286cf8e1)
Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.
Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
WorkloadMonitor resources are only needed when extra/monitoring is
deployed. Move them from system/monitoring to extra/monitoring package.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add migration 26 that re-labels monitoring resources to be owned by
monitoring-system HelmRelease instead of monitoring. This allows
seamless transition from direct resource management in extra/monitoring
to delegated management via system/monitoring chart.
Migration steps:
- Find all monitoring HelmReleases in tenant namespaces
- Suspend HelmRelease to prevent reconciliation
- Delete helm secrets to orphan resources
- Relabel all resources to monitoring-system
- Delete suspended HelmRelease
Also updates:
- monitoring-application.yaml: add monitoring-system component
- helmrelease.yaml: reference monitoring-system ExternalArtifact
- values.yaml: bump targetVersion to 25
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
What this PR does
This PR introduces the SLACK_SEVERITY_FILTER environment variable in the Alerta deployment to enable
filtering of alert severities for Slack notifications based on the disabledSeverity configuration.
Additionally, it adds a VMAgent resource template for scraping metrics within tenant namespaces, improving
monitoring granularity and control.
```release-note
[monitoring] Add SLACK_SEVERITY_FILTER for filtering Slack alert severities and VMAgent configuration for
tenant-specific metrics scraping.
```
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This patch introduces vmagent support inside tenant-level monitoring
stacks, ensuring that all tenant workloads have consistent and reliable
metrics scraping. By deploying a vmagent per tenant and wiring it to the
configured VictoriaMetrics short- and long-term storage, metrics are now
properly collected, labeled, and remote-written. This resolves gaps
where tenant environments previously missed metrics due to the absence
of dedicated agents.
```release-note
[monitoring,platform] Add vmagent integration into tenant monitoring
stacks to ensure complete and correct metrics collection for all tenants.
```
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This patch refactors the secret selectors to use the
`internal.cozystack.io/tenantresource` label for managing secret
visibility and removes any selectors based on it or the previous
`apps.cozystack.io/tenantresource` label, the idea being that this label
will only ever be set by the controller.
```
[controller,api] Refactor labels for the secret selector.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Since the VictoriaMetrics operator aggressively manages the metadata on
all owned components, the addition of labels by the lineage webhook
causes non-stop updates sent to the k8s API server. We mitigate this by
modifying the Monitoring Helm chart to set the `managedMetadata` field
on all VictoriaMetrics custom resources, where applicable.
```release-note
[monitoring] Explicitly set lineage labels on VictoriaMetrics' resources
known not to play nice when something modifies their owned resources in
flight.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Some k8s secrets created when deploying managed applications are
unhelpful to the end user or are outright not meant to be shown, because
they contain internal credentials not meant to be presented to the user.
This patch adds an `apps.cozystack.io/tenantresource=false` label to
such resources which will be later used to filter out such secrets in
the web UI.
```release-note
[platform] Mark non-user-facing secrets as such to avoid clutter in the
dashboard and leaking internal credentials.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>