Remove the custom Grafana image build now that the deployment runs on the
upstream grafana/grafana image with catalog plugins. Drops the Dockerfile,
the pinned image tag, the image targets in both monitoring Makefiles, and
the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana
is no longer produced or published.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Three package Makefiles declared an identical `test` target twice, so
every invocation printed "overriding commands for target 'test'" and
"ignoring old commands". Make names the lines in that warning: it was
already running the second recipe and discarding the first, so the first
declaration was dead in each case and removing it changes nothing but
the noise.
Which copy is dead is not the same in all three, so the edit is not
mechanical. In monitoring the `.PHONY: test` line sits above the dead
recipe rather than the live one; `.PHONY` applies to the target name
wherever it is declared, so the declaration stays and only the recipe
under it goes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The `OIDCUser` typedef in `packages/extra/monitoring/values.yaml`
declared only `@field {string} email`; `cozyvalues-gen` regenerated
`values.schema.json`, `README.md`, and the mirrored `openAPISchema`
on the `Monitoring` ApplicationDefinition WITHOUT a `role` field.
Consequences:
- The `@enum OIDCRole` declaration was orphan — referenced from
nowhere, so the schema's items object had `required: ["email"]`
and one property.
- A dashboard-driven CR would offer a form without any role picker;
a schema-conformant `{email}` submission would render into the
users-Job with `role: null`, `jq --raw-output` yields the string
`"null"`, the `POST /api/orgs/1/users` body carries
`"role":"null"`, Grafana returns HTTP 400, and the post-install
hook exhausts `backoffLimit: 6` — HelmRelease never becomes Ready.
- Unit + e2e tests all hardcoded `role: Admin` / `role: Viewer`, so
the contract mismatch never surfaced in CI.
Fix:
- Add `@field {OIDCRole} role - ...` to the `OIDCUser` typedef in
`packages/extra/monitoring/values.yaml` so cozyvalues-gen materialises
the field with enum + required.
- Regenerate `values.schema.json`, `README.md`, and mirrored
`openAPISchema` via `make generate`.
- Add `monitoring.oidc.assertUsersRoleShape` template-level guard
(same shape as `assertUsersEmailShape`) so a stale-schema CR or a
direct-helm invocation fails at render time with a legible error
instead of at hook execution with a Grafana 400.
- Consolidate all `spec.oidc.users` guards into a single
`monitoring.oidc.assertUsersPrologue` include and call it once from
`oidc-users-job.yaml`, ordering the incompat check
(`assertCustomSecretRefUsersEmpty`) before the per-entry role /
email regexes so operators see the more actionable message first.
Drop the duplicate include from `grafana.yaml` (Job template renders
in every mode anyway).
- Two new helm-unittest cases lock the regression:
`oidc.users entry missing role fails the render` and
`oidc.users entry with unknown role fails the render`.
- Point the previous `secretRef + users` incompat unit test at the
users-Job template since that's where the prologue now fires.
- `docs/oidc-grafana.md`: correct the failed-hook post-mortem window
claim — with `ttlSecondsAfterFinished: 3600` the Kubernetes TTL
controller reaps the Job + Pod 1h after the terminal condition,
not "until the next helm-upgrade".
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Previous docs claimed "the chart forces skip_org_role_sync = true
regardless of mode" — a promise the code did not keep in
CustomConfig mode. The chart is now fixed; sync the docs so the
guarantee reads truthfully and the incompatibility surface is
explicit:
- `docs/oidc-grafana.md`: rewrite the CustomConfig section so both
chart-forced settings are named explicitly, the `merge` semantics
are documented (operator can't override), and the
`secretRef` + `users` incompatibility is called out with an
explicit failure path.
- `system/monitoring/values.yaml`: expand the `config`/`secretRef`
comments so operators reading the values doc see the merge behavior
and the secretRef limitation before they hit the render-time fail.
Add `enabled: "true"` to the "typical keys" list — Grafana defaults
`enabled: false` and an inline map without it is inert.
- `extra/monitoring`: regenerated values.schema.json + README.md via
`make generate` (cozyvalues-gen + hack/update-crd.sh) to pick up
the `oidc.users[].email` doc-string touch; mirrored openAPISchema
on the ApplicationDefinition follows.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The chart previously injected the two settings the users-Job's
contract depends on — `skip_org_role_sync = true` and
`oauth_allow_insecure_email_lookup = true` — only inside the
`mode: System` branch of `grafana.yaml`. `values.yaml` and
`docs/oidc-grafana.md` both promised the settings would be forced
regardless of mode, but under `mode: CustomConfig` the operator's
inline `auth.generic_oauth` map was written verbatim, both
settings were absent, and the users-Job's role assignments were
silently stranded (no lookup by email → new Grafana account minted
at login) or transiently overwritten (no skip_org_role_sync → any
`role_attribute_path` or Grafana's `auto_assign_org_role: Viewer`
overwrote the Job's role on each login).
Fix: merge a `$chartForcedOidc` dict on top of both the generated
System block and the operator's inline map (`merge` wins for
overlapping keys, so the operator can NOT override the two
settings). CustomConfig secretRef mode cannot receive the merge
because the ini fragment is authoritative — the chart now hard-fails
render when both `customConfig.secretRef.name` and non-empty
`spec.oidc.users` are set, forcing the operator to either switch to
inline config or manage authorization themselves inside the ini.
Also:
- Align the inner HR `timeout` on `packages/extra/monitoring/templates/helmrelease.yaml`
from 10m to 15m so it matches the outer cozyrds
`release.cozystack.io/helm-install-timeout: "15m"` and does not
fail while the users-Job is still inside its
`activeDeadlineSeconds: 900`.
- Route the users-Job image through `cozy-lib.image` so
mirrored / air-gapped installations resolve it (matches the
pattern in `packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml`).
- Add `spec.ttlSecondsAfterFinished: 3600` on the users-Job so a
succeeded Job + Pod are garbage-collected on the next reconcile
interval rather than lingering until the next hook activation
(which never fires on release uninstall).
- Harden the users-Job shell:
* emails read NUL-terminated (`jq --raw-output0`) so an address
containing whitespace cannot corrupt the `for email in ...`
loop's field splitting;
* `/api/users/lookup?loginOrEmail=<email>` URL-encoded via
`jq @uri` so `&`/`=`/`?` in the local part parse correctly on
the Grafana side;
* prune loop switched from pipe-delimited fields to TSV inside
NUL-terminated records, with `cut -f` for extraction.
- Add a template-level assertion (`monitoring.oidc.assertUsersEmailShape`)
rejecting emails that do not match a conservative regex —
cheaper failure mode than a 400 from Grafana.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Extends the tenant-facing `Monitoring` CR with the new
`spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` field
consumed by the users-Job introduced in the previous commit. Works
for both `System` and `CustomConfig` modes; ignored when
`mode: None`.
Regenerated `packages/extra/monitoring/values.schema.json`,
`README.md`, and the mirrored `openAPISchema` on the
`monitoring-rd` ApplicationDefinition through the standard
`make generate` (cozyvalues-gen + hack/update-crd.sh) pipeline.
Also refreshed the stale annotation comment on the CR that still
claimed the chart provisions three KeycloakRealmGroups.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Column widths in the OIDC table contract by one character once the
tenant-facing `grafanaAdmin` row is gone. Regenerating the table
via `make generate` (cozyvalues-gen) so the pre-commit `make
generate` diff check in CI stays clean.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The tenant-facing `Monitoring` CR exposed `spec.oidc.grafanaAdmin`
via `packages/extra/monitoring/values.schema.json`, which the
ApplicationDefinition mirrored into `openAPISchema`. A tenant with
create/patch on `Monitoring` in their own namespace could set the
flag on their release, promoting members of their
`<clientId>-admin` Keycloak group to server-level `GrafanaAdmin`
instead of the intended org-level Admin.
Drop the field from the tenant surface. The platform bundle still
flips it on for the `monitoring-system` component through
`Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin`
(the platform Grafana chart continues to read the value verbatim
in `packages/system/monitoring/values.yaml`); tenant Grafana
releases inherit the chart default `false` and stay at org-level
Admin.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
Address the two blocking-class findings from the cozy-review pass on
this PR:
(1) Missing PackageSource dependency on cozystack.keycloak-operator
(MAJOR cold-install race).
Without this edge, the monitoring chart can reconcile before the
v1.edp.epam.com CRDs are registered on a fresh cluster — the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml then silently drops all
Keycloak-side objects for mode: System, and Grafana's
auth.generic_oauth block points at a client Keycloak never
provisions. Add the ordering edge, plus a
sources_monitoring_application_dependson_test.yaml regression
guard so a future edit cannot drop it silently.
(2) allow_assign_grafana_admin driven by `.Release.Name ==
monitoring-system` (MINOR fragile coupling).
Replace the release-name sniff with an explicit
`oidc.grafanaAdmin: false` values field, and flip it on for the
platform Grafana release via
Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
in packages/core/platform/templates/bundles/system.yaml —
mirroring the pattern monitoring-agents already uses for
per-component overrides. Tenant Grafana releases inherit the
chart default false and stay at org-level Admin.
Two new helm-unittest cases in packages/system/monitoring/tests/
lock the values-driven behaviour on both sides
(grafanaAdmin=true → Admin uplift, default → tenant stays at org
Admin), plus bundles_monitoring_grafana_admin_test.yaml in
packages/core/platform/tests/ regression-guards the Package's
component override so the platform release keeps its uplift on
future edits.
Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.
Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
packages/extra/monitoring/values.schema.json
packages/extra/monitoring/README.md
packages/system/monitoring-rd/cozyrds/monitoring.yaml
Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
The Monitoring public API surface (openAPISchema in monitoring-rd
cozyrds) is generated from packages/extra/monitoring/values.yaml by
cozyvalues-gen + hack/update-crd.sh — the foundation commit
(186239044) instead wrote it by hand in
packages/system/monitoring-rd/cozyrds/monitoring.yaml, which the
pre-commit hook then reverts against every push.
Move the oidc annotations into packages/extra/monitoring/values.yaml
(matching the pattern used by the rest of the top-level fields) and
run `make generate` in that directory. That regenerates:
- packages/extra/monitoring/values.schema.json
- packages/extra/monitoring/README.md
- packages/system/monitoring-rd/cozyrds/monitoring.yaml
(openAPISchema + keysOrder, both including the new oidc keys)
No behaviour change — the values.yaml under packages/system/monitoring
still carries the same oidc defaults so `helm template` works
stand-alone, and helm-unittest keeps passing (22/22).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
## What this PR does
Addresses #3181, where platform log ingestion could fail **silently**:
fluent-bit shipped to `vlinsert-generic` while the monitoring
`HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName`
in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did
not exist when the root tenant had monitoring disabled. Logs were
dropped with no failing signal.
Three complementary layers make the failure loud instead of silent:
1. **Readiness gate** —
`packages/extra/monitoring/templates/helmrelease.yaml` now sets a
`poller` `waitStrategy` and a `healthCheckExprs` entry on the
`VLCluster` `status.updateStatus`. A missing/unhealthy log store makes
the `monitoring-system` release **not-ready** instead of flipping
`Ready` the moment the CR is applied. Mirrors the existing seaweedfs
CNPG pattern.
2. **No empty-log-store hole** —
`packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render
on an empty `logsStorages`. With no entry no `VLCluster` exists, the
readiness gate has nothing to match, and fluent-bit would silently drop
logs — so this is now a loud misconfiguration.
3. **No silent fallback** — the `cozy-monitoring → tenant-root`
`vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are
created only when the root tenant hosts monitoring. The template reads
the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth,
so it cannot drift from the tenant's real state) and falls back to a new
platform value `monitoring.rootEnabled` (via
`_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first
render). The default preserves the historical always-create behaviour.
Covered by helm-unittest across all four affected charts (141 tests):
readiness-gate fields, empty-vs-populated `logsStorages` contract,
`ExternalName` present/absent fallback contract, and the platform wiring
into the `cozystack-values` Secret. Also adds the missing `test:` target
to `packages/extra/monitoring/Makefile` so its suite runs in CI.
**Known limitation / follow-up:** the legacy platform monitoring release
installed directly into `cozy-monitoring` via the `cozystack.monitoring`
PackageSource cannot carry the readiness gate, because the engine's
`ComponentInstall`/HelmRelease builder and the vendored
`helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`.
That source is not referenced by any active bundle; full coverage there
needs a helm-controller bump plus engine support.
### Release note
```release-note
fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a platform setting (`monitoring.rootEnabled`) to control whether
root-level monitoring is enabled.
* Monitoring redirect ExternalName services are now rendered only when
root monitoring is enabled (defaulting to enabled).
* **Bug Fixes**
* Ensures an explicitly set `monitoring.rootEnabled=false` is preserved
when generating platform configuration.
* Prevents incomplete monitoring deployments by failing rendering when
`logsStorages` is empty.
* Improves VictoriaMetrics readiness by delaying “ready” until the
operator reports an operational status.
* **Tests**
* Added/expanded Helm-unittest suites covering monitoring wiring,
redirects, readiness gating, and `logsStorages` requirements.
* Added a `make test` target to run chart unittests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on
PVCs only at creation — the StatefulSet controller never re-labels existing PVCs
and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster
(exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled
and the cleanup hook's selector misses them.
Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the
sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of
every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name
read from the CR's own spec.managedMetadata, so only monitoring's clusters are
touched (a user's own VMCluster carries no such label) and the value always
matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is
no worse than the pre-existing leak, so transient apiserver errors never abort the
platform upgrade. Bump platform targetVersion 49 -> 50.
Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is
never applied to PVCs by vm-operator at any current version, so the claim-template
label is the only mechanism — do not imply a v0.71.0 crossover.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Re-review on #3094 (IvanHunters): the AND selector kept managed-by=vm-operator,
which on the shipped vm-operator v0.68.4 is not reliably present on the storage
PVCs (it lives on the StatefulSet selector/pod labels, not the claim template),
making the delete fragile. Match by apps.cozystack.io/application.name=<release>-system
alone: that label is stamped on the VM/VL claim templates (67b087ab) and copied
onto the PVCs by IntoSTSVolume even on v0.68.4, is already release-scoped, and is
absent from the CNPG db PVCs (grafana-db/alerta-db carry only policy.cozystack.io
+ cnpg.io/*), so those stay untouched. Regression guard now forbids re-adding
managed-by to the selector.
Also apply the review's discipline items: the cleanup Job keeps only
before-hook-creation (a Failed run survives for post-mortem; hook-succeeded stays
on SA/Role/RoleBinding, cf. #3072) and sets backoffLimit: 0 (best-effort, no retry).
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added
in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The
apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster
spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto
PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091
storage leak stayed open.
Stamp the label where it actually reaches the PVCs: on
spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels
of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies
claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the
existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name=
<release>-system) now matches and stays scoped to this release. Verified by
rendering the CRs (label resolves to monitoring-system, matching the hook).
Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei
non-blocking) and a helm-unittest suite asserting the label lands on all three
storage claim templates. Corrected the misleading test comment about how the
label reaches the PVCs.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Address IvanHunters review on #3094: the PVC delete selector was
release-agnostic (-l managed-by=vm-operator), matching the VM storage
PVCs of every VMCluster/VLCluster in the namespace. If a namespace ever
hosted more than one monitoring release, tearing one down would delete
the other's VictoriaMetrics/VictoriaLogs storage.
Scope the selector to this release via apps.cozystack.io/application.name=
<release>-system, the ownership label the monitoring chart stamps on the
VM CRs' managedMetadata and vm-operator propagates onto the PVCs — mirroring
the merged seaweedfs hook's release scoping. deletecollection is still not
needed: kubectl delete -l lists then deletes individually.
Tests: assert the release-scoped selector and add a regression guard that
the delete never falls back to the bare managed-by=vm-operator selector.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Address review on #3094:
- Add --wait=false to both kubectl delete calls so PVC finalizers
(pvc-protection on pods still terminating) can never block the
post-delete hook and stall the HelmRelease teardown.
- Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop.
- Tests: assert the non-blocking flag, plus negative assertions that the
PVC delete stays label-scoped (no --all) and the secret delete stays
name-scoped (no -l/--all).
deletecollection RBAC verb intentionally not added: kubectl delete -l
lists then deletes individually, so the existing delete verb suffices.
BREAKING CHANGE: disabling monitoring now permanently removes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
When the monitoring tenant module is disabled, the vm-operator storage PVCs
(VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left
orphaned. Add a post-delete cleanup hook that deletes the PVCs by the
managed-by=vm-operator label and the two TLS secrets by name, plus a test
target and helm-unittest suite (the package had neither). CNPG PVCs are not
targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only
root fs, dropped caps, resource limits); failures are logged, never blocking
teardown.
NOTE: disabling monitoring now permanently removes its storage PVCs and the
data they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.
Three complementary changes:
- Readiness (do not stay silently Ready): the monitoring-system
HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
its VLCluster (status.updateStatus), so a missing/unhealthy log store
makes the release not-ready instead of flipping Ready as soon as the
CR is applied. Mirrors the seaweedfs CNPG pattern.
- No empty-log-store hole: system/monitoring rejects an empty
logsStorages at render time (helm fail). With no entry no VLCluster is
created, the readiness gate has nothing to match, and fluent-bit would
silently drop logs — so this is made a loud misconfiguration rather
than a documented footgun.
- No silent fallback: the cozy-monitoring -> tenant-root
vlinsert/vminsert ExternalName redirects in cozystack-basics are now
created only when the root tenant hosts monitoring, gated on
_cluster.monitoring-enabled — fed from a new platform value
monitoring.rootEnabled (default true, preserving the historical
always-create behaviour). monitoring.rootEnabled is the single
render-time source of truth: changing it re-renders cozystack-basics
(helm-controller watches the cozystack-values Secret it flows into). A
live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
cozystack-basics is an operator-generated HelmRelease with drift
detection off, so it re-renders only on install/upgrade/values-change
— never on a reconcile tick or when a lookup result changes — and a
lookup would go stale in both directions. Operators must keep
monitoring.rootEnabled in sync with Tenant/root spec.monitoring.
Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.
Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.
Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.
Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7e286cf8e1)
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Summary
- Adds `inlineScrapeConfig` support to the tenant `VMAgent` resource in
`packages/system/monitoring/templates/vm/vmagent.yaml`
- Adds commented usage example in both
`packages/system/monitoring/values.yaml` and
`packages/extra/monitoring/values.yaml`
## Problem
The tenant VMAgent resource does not support custom scrape
configurations. Using `additionalScrapeConfigs` (which references a
Kubernetes Secret) is not viable because tenant users have no access to
create or read Secrets — they can only manage resources under
`apps.cozystack.io`. Instead, `inlineScrapeConfig` accepts inline YAML
directly through Helm values, which is consistent with tenant
permissions.
Relates to #2194
## Usage
```yaml
vmagent:
inlineScrapeConfig: |
- job_name: "custom"
static_configs:
- targets: ["my-service:9090"]
```
## Test plan
- [ ] `helm template` monitoring with `inlineScrapeConfig` set — block
rendered
- [ ] `helm template` monitoring without it — no `inlineScrapeConfig` in
output
- [ ] Deploy and verify custom scrape targets are picked up by vmagent
```release-note
Add inlineScrapeConfig support to tenant vmagent
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Support for including an inline scrape configuration into VMAgent
monitoring when provided.
* **Documentation**
* Added a commented example showing how to supply an inline scrape job
and targets.
* Reordered remote-write URL entries in monitoring configuration for
clearer ordering.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.
Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Dashboard files should only be in system/monitoring.
The dashboards are created by system/monitoring when deployed
via HelmRelease from extra/monitoring.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
WorkloadMonitor resources are only needed when extra/monitoring is
deployed. Move them from system/monitoring to extra/monitoring package.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add migration 26 that re-labels monitoring resources to be owned by
monitoring-system HelmRelease instead of monitoring. This allows
seamless transition from direct resource management in extra/monitoring
to delegated management via system/monitoring chart.
Migration steps:
- Find all monitoring HelmReleases in tenant namespaces
- Suspend HelmRelease to prevent reconciliation
- Delete helm secrets to orphan resources
- Relabel all resources to monitoring-system
- Delete suspended HelmRelease
Also updates:
- monitoring-application.yaml: add monitoring-system component
- helmrelease.yaml: reference monitoring-system ExternalArtifact
- values.yaml: bump targetVersion to 25
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Fixes#749
## Changes
- Add documentation explaining how to enable Hubble for network
observability in Grafana
- Add four Grafana dashboards for Hubble metrics visualization
(Overview, DNS, L7 HTTP, Network)
- Register Hubble dashboards in the monitoring package
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced Hubble DNS Overview dashboard for namespace-level DNS
metrics visibility.
* Introduced Hubble L7 HTTP Metrics dashboard for HTTP request and
performance monitoring.
* Introduced Hubble Network Overview dashboard for network flows and
policy insights.
* **Documentation**
* Added comprehensive Hubble observability guide covering setup,
configuration, and troubleshooting.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add dedicated flux-tenants controller with label selector
--watch-label-selector=sharding.fluxcd.io/key=tenants to handle
tenant workloads separately from platform components.
Update all kubernetes app HelmReleases to:
- Use chartRef with ExternalArtifact instead of OCIRepository sourceRef
- Add sharding.fluxcd.io/key=tenants label
- Add cozystack.io/target-cluster-name label
Update fluxinstall to parse multiple YAML manifest files and
use flux service for storage-adv-addr.
Add cozystack-basics package for core tenant/namespace setup.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.
Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
(valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format
This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add documentation explaining how to enable Hubble for network
observability in Grafana. Include four pre-built Hubble dashboards
(overview, dns-namespace, l7-http-metrics, network-overview) and
register them in the monitoring hub's dashboard list.
Closes#749
Signed-off-by: majiayu000 <1835304752@qq.com>
What this PR does
This PR introduces the SLACK_SEVERITY_FILTER environment variable in the Alerta deployment to enable
filtering of alert severities for Slack notifications based on the disabledSeverity configuration.
Additionally, it adds a VMAgent resource template for scraping metrics within tenant namespaces, improving
monitoring granularity and control.
```release-note
[monitoring] Add SLACK_SEVERITY_FILTER for filtering Slack alert severities and VMAgent configuration for
tenant-specific metrics scraping.
```
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This patch introduces vmagent support inside tenant-level monitoring
stacks, ensuring that all tenant workloads have consistent and reliable
metrics scraping. By deploying a vmagent per tenant and wiring it to the
configured VictoriaMetrics short- and long-term storage, metrics are now
properly collected, labeled, and remote-written. This resolves gaps
where tenant environments previously missed metrics due to the absence
of dedicated agents.
```release-note
[monitoring,platform] Add vmagent integration into tenant monitoring
stacks to ensure complete and correct metrics collection for all tenants.
```
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This patch refactors the secret selectors to use the
`internal.cozystack.io/tenantresource` label for managing secret
visibility and removes any selectors based on it or the previous
`apps.cozystack.io/tenantresource` label, the idea being that this label
will only ever be set by the controller.
```
[controller,api] Refactor labels for the secret selector.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This patch implements name-based selectors for
`CozystackResourceDefinitions.spec.secrets`. Application developers may
now specify secrets that should or should not be visible to end users by
specifying a `resourceNames` field with a string slice of acceptable
names. This will, for instance, let developers exclude a secret like
`postgres-dbname-superuser` that has a predictable name even if it does
not have predictable labels. Simple templates are supported, so
`postgres-{{ .name }}-superuser` is also a valid entry under
`resourceNames`.
```release-note
[lineage, controller] Let application developers determine resource
visibility for end users by name, as well as by labels.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Since the VictoriaMetrics operator aggressively manages the metadata on
all owned components, the addition of labels by the lineage webhook
causes non-stop updates sent to the k8s API server. We mitigate this by
modifying the Monitoring Helm chart to set the `managedMetadata` field
on all VictoriaMetrics custom resources, where applicable.
```release-note
[monitoring] Explicitly set lineage labels on VictoriaMetrics' resources
known not to play nice when something modifies their owned resources in
flight.
```
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
[cozystack-controller] Introduce new dashboard-controller
[dashboard] Introduce new dashboard based on openapi-ui
Co-authored-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil