Remove the custom Grafana image build now that the deployment runs on the
upstream grafana/grafana image with catalog plugins. Drops the Dockerfile,
the pinned image tag, the image targets in both monitoring Makefiles, and
the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana
is no longer produced or published.
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Three package Makefiles declared an identical `test` target twice, so
every invocation printed "overriding commands for target 'test'" and
"ignoring old commands". Make names the lines in that warning: it was
already running the second recipe and discarding the first, so the first
declaration was dead in each case and removing it changes nothing but
the noise.
Which copy is dead is not the same in all three, so the edit is not
mechanical. In monitoring the `.PHONY: test` line sits above the dead
recipe rather than the live one; `.PHONY` applies to the target name
wherever it is declared, so the declaration stays and only the recipe
under it goes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Addresses #3181, where platform log ingestion could fail **silently**:
fluent-bit shipped to `vlinsert-generic` while the monitoring
`HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName`
in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did
not exist when the root tenant had monitoring disabled. Logs were
dropped with no failing signal.
Three complementary layers make the failure loud instead of silent:
1. **Readiness gate** —
`packages/extra/monitoring/templates/helmrelease.yaml` now sets a
`poller` `waitStrategy` and a `healthCheckExprs` entry on the
`VLCluster` `status.updateStatus`. A missing/unhealthy log store makes
the `monitoring-system` release **not-ready** instead of flipping
`Ready` the moment the CR is applied. Mirrors the existing seaweedfs
CNPG pattern.
2. **No empty-log-store hole** —
`packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render
on an empty `logsStorages`. With no entry no `VLCluster` exists, the
readiness gate has nothing to match, and fluent-bit would silently drop
logs — so this is now a loud misconfiguration.
3. **No silent fallback** — the `cozy-monitoring → tenant-root`
`vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are
created only when the root tenant hosts monitoring. The template reads
the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth,
so it cannot drift from the tenant's real state) and falls back to a new
platform value `monitoring.rootEnabled` (via
`_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first
render). The default preserves the historical always-create behaviour.
Covered by helm-unittest across all four affected charts (141 tests):
readiness-gate fields, empty-vs-populated `logsStorages` contract,
`ExternalName` present/absent fallback contract, and the platform wiring
into the `cozystack-values` Secret. Also adds the missing `test:` target
to `packages/extra/monitoring/Makefile` so its suite runs in CI.
**Known limitation / follow-up:** the legacy platform monitoring release
installed directly into `cozy-monitoring` via the `cozystack.monitoring`
PackageSource cannot carry the readiness gate, because the engine's
`ComponentInstall`/HelmRelease builder and the vendored
`helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`.
That source is not referenced by any active bundle; full coverage there
needs a helm-controller bump plus engine support.
### Release note
```release-note
fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a platform setting (`monitoring.rootEnabled`) to control whether
root-level monitoring is enabled.
* Monitoring redirect ExternalName services are now rendered only when
root monitoring is enabled (defaulting to enabled).
* **Bug Fixes**
* Ensures an explicitly set `monitoring.rootEnabled=false` is preserved
when generating platform configuration.
* Prevents incomplete monitoring deployments by failing rendering when
`logsStorages` is empty.
* Improves VictoriaMetrics readiness by delaying “ready” until the
operator reports an operational status.
* **Tests**
* Added/expanded Helm-unittest suites covering monitoring wiring,
redirects, readiness gating, and `logsStorages` requirements.
* Added a `make test` target to run chart unittests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
When the monitoring tenant module is disabled, the vm-operator storage PVCs
(VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left
orphaned. Add a post-delete cleanup hook that deletes the PVCs by the
managed-by=vm-operator label and the two TLS secrets by name, plus a test
target and helm-unittest suite (the package had neither). CNPG PVCs are not
targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only
root fs, dropped caps, resource limits); failures are logged, never blocking
teardown.
NOTE: disabling monitoring now permanently removes its storage PVCs and the
data they hold.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.
Three complementary changes:
- Readiness (do not stay silently Ready): the monitoring-system
HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
its VLCluster (status.updateStatus), so a missing/unhealthy log store
makes the release not-ready instead of flipping Ready as soon as the
CR is applied. Mirrors the seaweedfs CNPG pattern.
- No empty-log-store hole: system/monitoring rejects an empty
logsStorages at render time (helm fail). With no entry no VLCluster is
created, the readiness gate has nothing to match, and fluent-bit would
silently drop logs — so this is made a loud misconfiguration rather
than a documented footgun.
- No silent fallback: the cozy-monitoring -> tenant-root
vlinsert/vminsert ExternalName redirects in cozystack-basics are now
created only when the root tenant hosts monitoring, gated on
_cluster.monitoring-enabled — fed from a new platform value
monitoring.rootEnabled (default true, preserving the historical
always-create behaviour). monitoring.rootEnabled is the single
render-time source of truth: changing it re-renders cozystack-basics
(helm-controller watches the cozystack-values Secret it flows into). A
live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
cozystack-basics is an operator-generated HelmRelease with drift
detection off, so it re-renders only on install/upgrade/values-change
— never on a reconcile tick or when a lookup result changes — and a
lookup would go stale in both directions. Operators must keep
monitoring.rootEnabled in sync with Tenant/root spec.monitoring.
Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.
Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
PR #2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.
Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the #2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Every package Makefile tagged images via `settag` which, on a non-release
commit, collapsed to `:latest`. Concurrent PR builds therefore raced on every
image's `:latest` manifest in the OCIR registry. Two outliers (metallb,
redis-operator) used hardcoded version tags and collided across PRs even when
`settag` would not have.
Replace `settag` and the `TAG=latest` fallback with an `image-tags` macro
driven by three env vars:
- IMAGE_TAG — build-unique tag, always pushed
- PUBLISH_VERSIONED — also push :<component-version>
- PUBLISH_FLOATING — also push :latest
PR CI sets IMAGE_TAG=pr-<N>-<sha> with both PUBLISH_* off, so every push is
unique across runs. Release CI sets IMAGE_TAG=<git-tag> with both PUBLISH_*
on, preserving the previous release output and adding a uniform :<git-tag>
tag across every image. Local builds default to IMAGE_TAG=dev with PUSH=0
already off.
Image consumers reference digests pinned into values.yaml by each Makefile,
so the tag is decorative for cluster-side pulls; the build-unique tag is
sufficient for E2E.
Drops the redundant `:<component-version>-<git-tag>` form (cilium, linstor,
matchbox, ubuntu-container-disk, kubevirt-*, cluster-autoscaler) — nothing
in the repo or downstream values.yaml consumed it. The same information is
preserved via the new uniform `:<git-tag>` plus the existing
`:<component-version>` on release.
ubuntu-container-disk builds a separate image per Kubernetes version in a
loop. Switched its primary tag to `:<k8s-ver>-<IMAGE_TAG>` so loop
iterations within a single build do not push to the same tag.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
[cozystack-controller] Introduce new dashboard-controller
[dashboard] Introduce new dashboard based on openapi-ui
Co-authored-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Use https://github.com/cozystack/cozyvalues-gen for three apps:
- apps/postgres
- apps/virtual-machine
- extra/monitoring
Changes:
- Add type and enum definitions to values.yaml.
- Update READMEs with new information.
- Update values.schema.json with definitions for children objects,
allowing precise UI customization. Add regexp for specific types
such as resources: CPU like `500m` and RAM like `4GiB`.
- Remove direct injections with `yq` from Makefiles where they're not
needed anymore.
Co-authored-by: Nick Volynkin <nick.volynkin@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Updated Grafana to version 11.4.0
- Added new Grafana plugins: VictoriaMetrics logs datasource, Natel
Discrete Panel, and Worldmap Panel
- **Improvements**
- Enhanced Grafana image build process
- Dynamically manage Grafana image versioning
- Updated plugin installation method
- **Version Update**
- Monitoring package version bumped from 1.7.0 to 1.8.0
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
This change is aimed at improving the development experience.
- The option `make delete` has been added.
- Added check for `NAME` and `NAMESPACE` variables
- Now, any package (not just system ones) can include options such as
make show, make diff, make apply.
- Applications from packages/extra require explicit specification of the
`NAMESPACE`.
- Applications from packages/apps require explicit specification of both
`NAME` and `NAMESPACE`.
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil